Oct 31 18:20:19.222888 ip-10-0-132-99 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 18:20:19.620253 ip-10-0-132-99 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:19.620253 ip-10-0-132-99 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 18:20:19.620253 ip-10-0-132-99 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:19.620253 ip-10-0-132-99 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:19.624712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624628    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:19.624712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624709    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:19.624712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624714    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:19.624712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624719    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624725    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624729    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624732    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624736    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624740    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624744    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624748    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624752    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624756    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624760    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624764    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624767    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624771    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624775    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624778    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624783    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624791    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624796    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624800    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:19.624961 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624804    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624808    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624811    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624815    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624819    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624823    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624827    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624831    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624835    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624839    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624844    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624848    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624853    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624857    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624861    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624866    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624870    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624875    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624879    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624883    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:19.625775 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624887    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624891    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624899    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624905    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624910    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624914    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624919    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624923    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624927    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624932    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624937    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624942    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624946    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624950    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624954    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624958    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624962    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624966    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624970    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:19.626573 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624974    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624979    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624983    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624988    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624992    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.624997    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625001    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625006    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625010    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625015    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625019    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625023    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625027    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625031    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625035    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625039    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625043    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625047    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625061    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:19.627374 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625066    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625070    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625075    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625079    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625084    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625088    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625092    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625096    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625100    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625104    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625108    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625112    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625117    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625121    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625125    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625130    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625134    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625139    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625144    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:19.628236 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625148    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625154    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625162    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625167    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625172    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.625177    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626504    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626516    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626520    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626525    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626529    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626534    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626538    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626542    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626546    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626551    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626555    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626559    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626564    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626568    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:19.628940 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626572    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626594    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626599    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626603    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626608    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626611    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626616    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626620    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626625    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626630    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626634    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626639    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626643    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626647    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626651    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626656    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626660    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626664    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626668    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:19.629446 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626672    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626676    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626680    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626684    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626689    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626695    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626703    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626708    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626714    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626718    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626723    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626727    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626732    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626736    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626740    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626745    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626749    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626753    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626757    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:19.630024 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626761    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626766    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626771    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626776    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626780    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626784    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626788    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626792    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626797    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626801    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626805    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626810    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626814    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626818    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626822    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626826    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626830    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626834    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626841    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:19.630498 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626846    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626851    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626856    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626861    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626866    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626870    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626874    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626879    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626884    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626889    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626893    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626897    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626901    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626906    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626910    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626914    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626918    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626923    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626928    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626932    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:19.630952 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626937    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626941    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626945    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626951    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626955    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626959    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626963    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626968    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626972    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626977    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626981    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626985    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626989    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626993    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.626997    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628041    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628054    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628058    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628063    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:19.631420 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628083    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628090    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628095    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628100    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628105    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628110    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628115    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628135    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628140    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628145    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628149    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628154    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628158    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628163    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628167    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628172    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628176    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628181    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628185    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628190    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:19.631899 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628199    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628204    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628208    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628213    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628218    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628222    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628226    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628231    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628235    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628239    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628244    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628248    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628257    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628264    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628271    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628276    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628280    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628285    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628290    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628294    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:19.632367 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628298    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628303    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628308    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628312    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628322    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628326    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628330    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628335    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628339    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628344    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628350    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628356    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628361    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628366    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628370    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628375    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628379    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628388    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628392    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:19.632860 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628397    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628401    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628405    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628410    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628414    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628419    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628425    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628431    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628435    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628440    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628444    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628454    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628458    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628462    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628466    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628471    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628476    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628480    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628485    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:19.633294 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628490    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628494    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628499    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628503    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628512    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628517    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628522    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628526    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628531    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628535    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628539    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628543    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628548    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628552    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628556    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628561    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628565    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628594    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628599    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628606    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:19.633760 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628610    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628615    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628623    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.628627    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629762    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629783    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629791    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629796    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629801    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629805    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629809    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629813    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629817    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629820    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629823    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629826    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629830    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629833    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629835    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629838    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629841    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629844    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629849    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 31 18:20:19.634214 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629851    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629855    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629858    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629861    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629865    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629868    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629872    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629875    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629879    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629881    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629884    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629888    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629890    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629895    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629898    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629902    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629904    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629907    2571 flags.go:64] FLAG: --enable-server="true"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629910    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629920    2571 flags.go:64] FLAG: --event-burst="100"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629924    2571 flags.go:64] FLAG: --event-qps="50"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629927    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629930    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629933    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 31 18:20:19.634772 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629937    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629940    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629944    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629947    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629950    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629952    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629955    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629958    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629961    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629964    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629966    2571 flags.go:64] FLAG: --feature-gates=""
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629975    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629978    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629982    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629985    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629988    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629991    2571 flags.go:64] FLAG: --help="false"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629993    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.629997    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630000    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630003    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630007    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630011    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 18:20:19.635359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630014    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630017    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630020    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630022    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630026    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630029    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630032    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630035    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630038    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630041    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630044    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630047    2571 flags.go:64] FLAG: --lock-file=""
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630050    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630053    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630056    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630065    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630068    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630071    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630074    2571 flags.go:64] FLAG: --logging-format="text"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630077    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630080    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630083    2571 flags.go:64] FLAG: --manifest-url=""
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630086    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630091    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630094    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 18:20:19.635929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630097    2571 flags.go:64] FLAG: --max-pods="110"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630100    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630103    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630106    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630109    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630112    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630115    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630118    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630127    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630130    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630133    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630137    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630139    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630142    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630145    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630148    2571 flags.go:64] FLAG: --port="10250"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630151    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630156    2571 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b6db748a16ba5b95"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630159    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630162    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630165    2571 flags.go:64] FLAG: --register-node="true"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630168    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630171    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630174    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630177    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630180    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 31 18:20:19.636512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630184    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630187    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630190    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630193    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630196    2571 flags.go:64] FLAG: --runonce="false"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630201    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630204    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630207    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630210    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630212    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630216    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630219    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630221    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630224    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630227    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630230    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630232    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630235    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630238    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630241    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630247    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630250    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630253    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630258    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630261    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 18:20:19.637182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630264    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630266    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630269    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630272    2571 flags.go:64] FLAG: --v="2"
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630277    2571 flags.go:64] FLAG: --version="false"
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630281    2571 flags.go:64] FLAG: --vmodule=""
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630285    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630288    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630427    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630432    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630435    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630438    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630442    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630445    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630448    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630450    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630453    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630455    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630458    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630461    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630463    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:19.637795 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630466    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630468    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630471    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630473    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630476    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630478    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630481    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630483    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630485    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630489    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630492    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630495    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630498    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630501    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630503    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630506    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630509    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630512    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630514    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:19.638280 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630517    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630519    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630522    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630524    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630529    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630531    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630533    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630536    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630538    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630541    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630543    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630546    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630548    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630550    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630553    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630555    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630558    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630560    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630564    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:19.638751 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630567    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630570    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630573    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630592    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630597    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630601    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630605    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630613    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630616    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630618    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630620    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630623    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630625    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630628    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630630    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630633    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630635    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630639    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630641    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:19.639235 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630644    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630646    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630649    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630652    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630654    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630657    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630659    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630661    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630664    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630666    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630669    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630671    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630674    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630676    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630678    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630681    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630683    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630686    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630688    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:19.639712 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630691    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630693    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630701    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630704    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630707    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630709    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630711    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630714    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630716    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630719    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630721    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630725    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630727    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630730    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630732    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630734    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:19.640156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630737    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.630765    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630801    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630806    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630809    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630812    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630814    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630817    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630819    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630822    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630824    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630827    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630829    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630832    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630834    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630837    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:19.640560 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630840    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630842    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630844    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630851    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630854    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630856    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630859    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630862    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630866    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630869    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630872    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630875    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630878    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630880    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630883    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630885    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630888    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630890    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630893    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:19.640938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630895    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630898    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630900    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630902    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630904    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630907    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630909    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630911    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630914    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630917    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630920    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630922    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630925    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630927    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630930    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630932    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630934    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630937    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630940    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:19.641385 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630942    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630944    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630947    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630949    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630951    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630956    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630958    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630961    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630963    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630966    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630969    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630971    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630974    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630976    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630978    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630981    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630984    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630986    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630988    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630991    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:19.641880 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630993    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630996    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.630999    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631001    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631004    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631008    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631012    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631014    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631017    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631020    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631022    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631025    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631028    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631030    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631033    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631036    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631038    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631042    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631044    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:19.642341 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631047    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631049    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631051    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631054    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631056    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631059    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631061    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631063    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631066    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631068    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631071    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631074    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631077    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631079    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.631081    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.638464    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.638481    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.638494    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.638499    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 18:20:19.642929 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638555    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638560    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638563    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638566    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638569    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638572    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638574    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638591    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638593    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638596    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638599    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638602    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638604    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638607    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638610    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638612    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638614    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638617    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638620    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638622    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:19.643381 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638625    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638627    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638630    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638632    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638635    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638637    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638640    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638642    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638644    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638649    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638652    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638654    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638657    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638659    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638662    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638664    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638667    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638669    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638672    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:19.644105 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638676    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638680    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638683    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638686    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638688    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638691    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638693    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638696    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638699    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638701    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638704    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638706    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638709    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638711    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638713    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638716    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638720    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638724    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638726    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:19.644826 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638729    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638733    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638736    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638739    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638742    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638745    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638748    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638750    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638753    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638756    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638758    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638761    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638763    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638766    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638769    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638771    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638774    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638777    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638779    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:19.645310 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638782    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638785    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638787    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638790    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638793    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638795    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638797    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638800    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638802    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638805    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638807    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638810    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638813    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638815    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638818    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638820    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638823    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638825    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638828    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638831    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:19.645925 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638834    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638836    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638838    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638841    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638843    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638846    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638848    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638850    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.638853    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.638880    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639028    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639033    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639036    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639039    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639042    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639046    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:19.646528 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639048    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639051    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639053    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639056    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639058    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639061    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639063    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639066    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639068    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639071    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639073    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639075    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639078    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639080    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639082    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639091    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639094    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639097    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639099    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639102    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:19.646977 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639104    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639107    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639109    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639112    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639114    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639117    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639119    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639121    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639124    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639126    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639129    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639131    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639135    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639138    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639141    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639144    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639146    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639149    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639151    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:19.647454 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639154    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639156    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639159    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639161    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639164    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639166    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639169    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639171    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639174    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639179    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639182    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639184    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639187    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639189    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639192    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639194    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639196    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639199    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639201    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:19.647989 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639204    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639206    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639209    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639211    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639214    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639216    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639218    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639221    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639223    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639226    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639228    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639231    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639233    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639235    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639238    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639241    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639243    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639246    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639248    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:19.648958 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639252    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639256    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639258    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639261    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639265    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639268    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639270    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639273    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639275    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639278    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639280    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639283    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639285    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639287    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639290    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639292    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639295    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639297    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639299    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:19.649458 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639301    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639304    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639306    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639309    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.639331    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639361    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639364    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639367    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639370    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639372    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639375    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639377    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639379    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639382    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639384    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639386    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:19.650013 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639389    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639391    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639394    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639396    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639399    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639403    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639406    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639408    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639410    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639413    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639415    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639418    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639420    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639422    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639424    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639427    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639429    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639432    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639434    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:19.650402 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639437    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639440    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639444    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639447    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639449    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639452    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639454    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639457    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639459    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639462    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639464    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639466    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639469    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639471    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639474    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639477    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639480    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639482    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639486    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:19.650888 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639489    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639491    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639493    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639503    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639506    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639509    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639511    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639514    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639516    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639519    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639522    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639524    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639527    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639529    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639532    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639534    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639537    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639539    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639542    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639544    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:19.651389 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639546    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639549    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639551    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639554    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639557    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639559    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639562    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639566    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639569    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639572    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639575    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639595    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639600    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639603    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639605    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639608    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639610    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639613    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639616    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:19.651938 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639618    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639621    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639623    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639626    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639628    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639631    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639633    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639636    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639638    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639641    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639643    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639645    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639648    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639650    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639654    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639656    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639658    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:19.639661    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.639719    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.641527    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 18:20:19.652477 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.642488    2571 server.go:1014] "Starting client certificate rotation"
Oct 31 18:20:19.652996 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.642612    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 18:20:19.652996 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.642661    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 18:20:19.652996 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.644839    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 18:20:19.652996 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.647618    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 18:20:19.659895 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.659882    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 31 18:20:19.662709 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.662694    2571 log.go:25] "Validated CRI v1 image API"
Oct 31 18:20:19.663493 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.663468    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 18:20:19.666121 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.666104    2571 fs.go:135] Filesystem UUIDs: map[6044eb23-9831-4de4-8c85-af58aa0d139f:/dev/nvme0n1p4 710be49e-b8cb-4e11-916d-abe5e7617268:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 31 18:20:19.666176 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.666122    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 18:20:19.671933 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.671817    2571 manager.go:217] Machine: {Timestamp:2026-10-31 18:20:19.670557396 +0000 UTC m=+0.331100696 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086560 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25912fb3436316d6440ea18661b548 SystemUUID:ec25912f-b343-6316-d644-0ea18661b548 BootID:c3512f76-0f79-4456-b4c5-0080dfff8e53 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:f0:bd:22:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:f0:bd:22:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:b9:7d:e8:e3:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 18:20:19.672261 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.672250    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 18:20:19.672339 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.672328    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 18:20:19.675637 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.675608    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 18:20:19.675771 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.675640    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-99.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 18:20:19.675811 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.675773    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 18:20:19.675811 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.675782    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 18:20:19.675811 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.675808    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 18:20:19.675889 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.675820    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 18:20:19.676513 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.676504    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 18:20:19.679401 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.679389    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 18:20:19.679889 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.679879    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 18:20:19.680550 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.680541    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 18:20:19.680589 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.680554    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 18:20:19.680589 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.680564    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 18:20:19.680589 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.680572    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 31 18:20:19.680715 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.680591    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 18:20:19.680928 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.680915    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 18:20:19.682961 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.682946    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 18:20:19.687464 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.687448    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 18:20:19.688195 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688181    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 18:20:19.688260 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688212    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 18:20:19.688374 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688364    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 18:20:19.688405 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688387    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 18:20:19.688405 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688393    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 18:20:19.688405 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688398    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 18:20:19.688405 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688404    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 18:20:19.688507 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688410    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 18:20:19.688868 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688859    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 18:20:19.688868 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688868    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 18:20:19.688933 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688882    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 18:20:19.688933 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688888    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 18:20:19.688933 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688900    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 18:20:19.688933 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.688909    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 18:20:19.690202 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.690192    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 18:20:19.690202 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.690202    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 18:20:19.695636 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.695623    2571 server.go:1263] "Started kubelet"
Oct 31 18:20:19.695731 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.695703    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 18:20:19.695848 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.695713    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 18:20:19.695915 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.695874    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 18:20:19.696421 ip-10-0-132-99 systemd[1]: Started Kubernetes Kubelet.
Oct 31 18:20:19.696991 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.696969    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 18:20:19.698040 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.698027    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 18:20:19.699491 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.699472    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 18:20:19.703884 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.703868    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 18:20:19.704255 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.704238    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 18:20:19.704639 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.704619    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-99.us-west-1.compute.internal\" not found"
Oct 31 18:20:19.705018 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705000    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 18:20:19.705018 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705003    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 18:20:19.705144 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705030    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 18:20:19.705144 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705100    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 18:20:19.705144 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705108    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 18:20:19.705230 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705190    2571 factory.go:55] Registering systemd factory
Oct 31 18:20:19.705230 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705218    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 31 18:20:19.705431 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705418    2571 factory.go:153] Registering CRI-O factory
Oct 31 18:20:19.705470 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705434    2571 factory.go:223] Registration of the crio container factory successfully
Oct 31 18:20:19.705498 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705473    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 18:20:19.705498 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705487    2571 factory.go:103] Registering Raw factory
Oct 31 18:20:19.705498 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.705497    2571 manager.go:1199] Started watching for new ooms in manager
Oct 31 18:20:19.706892 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.706877    2571 manager.go:319] Starting recovery of all containers
Oct 31 18:20:19.718342 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.718325    2571 manager.go:324] Recovery completed
Oct 31 18:20:19.722783 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.722771    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:19.727309 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.727290    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:19.727375 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.727322    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:19.727375 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.727334    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:19.727825 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.727813    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 31 18:20:19.727825 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.727824    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 18:20:19.727922 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.727838    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 18:20:19.727922 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.727867    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:19.728647 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.728628    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:19.728716 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.728660    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:19.728716 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.728671    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:19.729017 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.729004    2571 policy_none.go:50] "Start"
Oct 31 18:20:19.729062 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.729040    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 18:20:19.729102 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.729070    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 18:20:19.730034 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.730023    2571 policy_none.go:44] "Start"
Oct 31 18:20:19.775955 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.772223    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 18:20:19.777387 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.777238    2571 manager.go:341] "Starting Device Plugin manager"
Oct 31 18:20:19.777387 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.777276    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 18:20:19.777387 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.777289    2571 server.go:87] "Starting device plugin registration server"
Oct 31 18:20:19.777601 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.777572    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 18:20:19.777662 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.777605    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 18:20:19.777811 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.777790    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 18:20:19.777989 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.777898    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 18:20:19.777989 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.777913    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 18:20:19.778390 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.778370    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 18:20:19.778462 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.778443    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.us-west-1.compute.internal\" not found"
Oct 31 18:20:19.787144 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.787113    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 18:20:19.788513 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.788499    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 18:20:19.788623 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.788521    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 18:20:19.788623 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.788543    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 18:20:19.788623 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.788599    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 18:20:19.835869 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.835842    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 18:20:19.835998 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.835893    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 18:20:19.836744 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.835832    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.us-west-1.compute.internal.18e3b03552fc154f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 18:20:19.695596879 +0000 UTC m=+0.356140187,LastTimestamp:2026-10-31 18:20:19.695596879 +0000 UTC m=+0.356140187,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.us-west-1.compute.internal,}"
Oct 31 18:20:19.878173 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.878126    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:19.878982 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.878966    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:19.879069 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.878994    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:19.879069 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.879005    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:19.879069 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.879028    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:19.888695 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.888676    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"]
Oct 31 18:20:19.888793 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.888740    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:19.889432 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.889418    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:19.889506 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.889448    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:19.889506 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.889459    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:19.890728 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.890714    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:19.890857 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.890842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:19.890914 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.890880    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:19.891346 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.891330    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:19.891435 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.891364    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:19.891435 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.891330    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:19.891435 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.891404    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:19.891435 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.891420    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:19.891435 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.891376    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:19.892442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.892428    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:19.892492 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.892455    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:19.893082 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.893069    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:19.893148 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.893093    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:19.893148 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.893103    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:19.915073 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.915046    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.us-west-1.compute.internal\" not found" node="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:19.918868 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.918851    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 18:20:19.918953 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.918889    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qcl97"
Oct 31 18:20:19.920630 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:19.920615    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.us-west-1.compute.internal\" not found" node="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:19.990482 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.990463    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:19.991908 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:19.991894    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:20.006938 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.006918    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/401f601c1d615f3dbfc663ac1fbee6e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"401f601c1d615f3dbfc663ac1fbee6e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.007012 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.006944    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.007012 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.006965    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.089806 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.089773    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:20.090272 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.090247    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.090272 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:20.090271    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-99.us-west-1.compute.internal\": node \"ip-10-0-132-99.us-west-1.compute.internal\" not found"
Oct 31 18:20:20.090762 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.090746    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:20.104955 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.104940    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.107151 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.107137    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/401f601c1d615f3dbfc663ac1fbee6e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"401f601c1d615f3dbfc663ac1fbee6e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.107193 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.107160    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.107193 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.107178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.107253 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.107229    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.107284 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.107245    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/401f601c1d615f3dbfc663ac1fbee6e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"401f601c1d615f3dbfc663ac1fbee6e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.107284 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.107249    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.164615 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.164544    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:20:20.165806 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.165794    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.217763 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.217738    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.223248 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.223230    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:20:20.223529 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.223514    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:20:20.642744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.642672    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 18:20:20.643276 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.642806    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 18:20:20.643276 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.642807    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 18:20:20.643276 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.642824    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 18:20:20.643276 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:20.642825    2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7971b018aa4c4793a97432a2228923c-30a6b095e3cfe1a3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-132-99.us-west-1.compute.internal.18e3b03554dff116\": read tcp 10.0.132.99:41324->3.133.177.27:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-132-99.us-west-1.compute.internal.18e3b03554dff116  default   5349 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 18:20:19 +0000 UTC,LastTimestamp:2026-10-31 18:20:19.889432076 +0000 UTC m=+0.549975380,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.us-west-1.compute.internal,}"
Oct 31 18:20:20.681636 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.681617    2571 apiserver.go:52] "Watching apiserver"
Oct 31 18:20:20.704309 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.704288    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 18:20:20.923857 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.923779    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 18:15:21 +0000 UTC" deadline="2028-04-06 01:47:53.608793292 +0000 UTC"
Oct 31 18:20:20.923857 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.923806    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12535h27m32.684990341s"
Oct 31 18:20:20.945018 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.944997    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 18:20:20.945324 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.945304    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc","openshift-image-registry/node-ca-rlwgx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-rg4dh","openshift-multus/multus-bx85h","openshift-multus/network-metrics-daemon-mgb86","openshift-network-diagnostics/network-check-target-ztv9x","kube-system/global-pull-secret-syncer-m56gf","kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal","openshift-dns/node-resolver-xjpst","openshift-network-operator/iptables-alerter-25sgx","kube-system/konnectivity-agent-hgxrq","openshift-cluster-node-tuning-operator/tuned-4g2hb","openshift-ovn-kubernetes/ovnkube-node-qtzdz"]
Oct 31 18:20:20.946727 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.946692    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:20.947736 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.947720    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:20.948676 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.948653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:20.949637 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.949621    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bx85h"
Oct 31 18:20:20.950646 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.950633    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:20.950729 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:20.950698    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:20.951645 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.951625    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:20.951726 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:20.951683    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:20.952566 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.952553    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:20.952659 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:20.952606    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:20.954011 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.953996    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:20.956358 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.956300    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:20.957291 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.957277    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:20.958354 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.958335    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:20.959535 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.959518    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:20.998575 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.998549    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 18:20:20.998575 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.998552    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 18:20:20.998780 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.998608    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgxxf\""
Oct 31 18:20:20.998780 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.998598    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 18:20:20.998780 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.998693    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 18:20:20.999071 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.999057    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 18:20:20.999145 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:20.999128    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-szhfk\""
Oct 31 18:20:21.000349 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.000334    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 18:20:21.006551 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.006534    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 18:20:21.012109 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-k8s-cni-cncf-io\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.012179 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-kubernetes\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.012179 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012156    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-tuned\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.012266 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.012266 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012223    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:21.012266 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012251    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.012383 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012298    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-tmp\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.012383 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012333    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-var-lib-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012383 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012367    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-ovn\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012401    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-config\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-cni-binary-copy\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.012512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012454    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-var-lib-kubelet\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.012512 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-kubelet\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012512    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-slash\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012530    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-env-overrides\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012558    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-etc-selinux\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.012659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-multus\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.012659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-modprobe-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.012659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/a8e2f392-7651-4d4d-9c09-b4978aabcebf-kube-api-access-ldl6f\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.012659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-netns\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.012659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012654    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-kubelet-config\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012677    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-systemd\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-lib-modules\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012712    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-etc-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012726    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-script-lib\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012749    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012799    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-registration-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012871    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-conf-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12b631c1-447d-43ee-9f5a-d77bfb04e03c-hosts-file\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-node-log\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012939    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-cnibin\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-system-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.012967 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012974    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.012997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czvds\" (UniqueName: \"kubernetes.io/projected/1fe78dfe-d1f0-4196-9832-702532afcaf8-kube-api-access-czvds\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013016    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-sys\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-system-cni-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtb7n\" (UniqueName: \"kubernetes.io/projected/3386381a-8b41-473b-9400-04d3d3eb2c7e-kube-api-access-mtb7n\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013070    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-sys-fs\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-kubelet\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013129    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-bin\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a065905-b86e-4326-a7d8-ec336cb938c7-ovn-node-metrics-cert\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013155    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e2f392-7651-4d4d-9c09-b4978aabcebf-host\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013169    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2efd8852-abb6-40d2-afda-40fe91bbd18f-agent-certs\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-socket-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013233    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-multus-certs\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-log-socket\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.013461 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013319    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcqf\" (UniqueName: \"kubernetes.io/projected/ec058e17-0198-4c81-8ce4-2e3a616f9d69-kube-api-access-jkcqf\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013342    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013373    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrt52\" (UniqueName: \"kubernetes.io/projected/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kube-api-access-qrt52\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013397    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysconfig\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013425    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-run\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-systemd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013474    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2efd8852-abb6-40d2-afda-40fe91bbd18f-konnectivity-ca\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-dbus\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013544    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmw9b\" (UniqueName: \"kubernetes.io/projected/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-kube-api-access-mmw9b\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3497b74a-87c3-4f27-91b7-dac04680a875-host-slash\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013600    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-cnibin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013627    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-host\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013646    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-device-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-socket-dir-parent\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013676    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b631c1-447d-43ee-9f5a-d77bfb04e03c-tmp-dir\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.014023 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013690    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpzgw\" (UniqueName: \"kubernetes.io/projected/12b631c1-447d-43ee-9f5a-d77bfb04e03c-kube-api-access-mpzgw\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013704    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-netd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013719    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3497b74a-87c3-4f27-91b7-dac04680a875-iptables-alerter-script\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5tp\" (UniqueName: \"kubernetes.io/projected/3497b74a-87c3-4f27-91b7-dac04680a875-kube-api-access-6m5tp\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-bin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013793    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-etc-kubernetes\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013812    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013829    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-daemon-config\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013849    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-netns\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-hostroot\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-conf\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013912    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-systemd-units\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013969    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cprjg\" (UniqueName: \"kubernetes.io/projected/0a065905-b86e-4326-a7d8-ec336cb938c7-kube-api-access-cprjg\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.014442 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.013996    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8e2f392-7651-4d4d-9c09-b4978aabcebf-serviceca\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.096590 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.096566    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 18:20:21.096655 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.096647    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wqj2b"
Oct 31 18:20:21.098603 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.098567    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 18:20:21.098689 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.098672    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 18:20:21.098900 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.098884    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 18:20:21.098964 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.098906    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w82k4\""
Oct 31 18:20:21.099266 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.099246    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 18:20:21.100300 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.100280    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m5pkq\""
Oct 31 18:20:21.100395 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.100327    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:20:21.100395 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.100378    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:20:21.100539 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.100526    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 18:20:21.100669 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.100654    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 18:20:21.101054 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101024    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 18:20:21.101124 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101073    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 18:20:21.101241 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101227    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 18:20:21.101317 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101305    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 18:20:21.101416 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101402    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 18:20:21.101513 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101500    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 18:20:21.101660 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101641    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 18:20:21.101766 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101753    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 18:20:21.101985 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101966    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xr757\""
Oct 31 18:20:21.102056 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.101969    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 18:20:21.102174 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.102157    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72cf5\""
Oct 31 18:20:21.102328 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.102316    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4kdr\""
Oct 31 18:20:21.102475 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.102455    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 18:20:21.102475 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.102464    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 18:20:21.111707 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.111687    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9xfhz\""
Oct 31 18:20:21.111806 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.111708    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 18:20:21.111806 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.111715    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 18:20:21.111806 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.111710    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-27pxv\""
Oct 31 18:20:21.114664 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprjg\" (UniqueName: \"kubernetes.io/projected/0a065905-b86e-4326-a7d8-ec336cb938c7-kube-api-access-cprjg\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.114747 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114674    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8e2f392-7651-4d4d-9c09-b4978aabcebf-serviceca\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.114747 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114698    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-k8s-cni-cncf-io\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.114747 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114722    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-kubernetes\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.114747 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114744    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-tuned\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.114938 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114781    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-k8s-cni-cncf-io\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.114938 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114808    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-kubernetes\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.114938 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.114938 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:21.114938 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.114938 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-tmp\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.114938 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-var-lib-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-ovn\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114971    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-config\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.114979    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.114992    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-cni-binary-copy\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115013    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-var-lib-kubelet\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.115037    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret podName:28594e99-2b75-44a6-8922-315e6b680700 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:21.615017435 +0000 UTC m=+2.275560737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret") pod "global-pull-secret-syncer-m56gf" (UID: "28594e99-2b75-44a6-8922-315e6b680700") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115042    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115063    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-kubelet\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115078    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-ovn\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115088    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-slash\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115065    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-var-lib-kubelet\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115134    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-kubelet\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115136    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115151    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-var-lib-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115152    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8e2f392-7651-4d4d-9c09-b4978aabcebf-serviceca\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115184    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-env-overrides\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-etc-selinux\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.115320 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115227    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-slash\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115243    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-multus\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115266    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-modprobe-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/a8e2f392-7651-4d4d-9c09-b4978aabcebf-kube-api-access-ldl6f\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-netns\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115342    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-kubelet-config\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-systemd\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115377    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-etc-selinux\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-lib-modules\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115423    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-etc-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115429    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-script-lib\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115475    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-lib-modules\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115482    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-etc-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115540    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-netns\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115601    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-multus\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115645    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.116007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115654    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-env-overrides\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115703    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-modprobe-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115705    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-kubelet-config\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115720    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-config\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115744    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-systemd\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115784    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-cni-binary-copy\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115791    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-registration-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115838    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-conf-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12b631c1-447d-43ee-9f5a-d77bfb04e03c-hosts-file\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.115876    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115893    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-node-log\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115907    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-conf-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.115921    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs podName:ec058e17-0198-4c81-8ce4-2e3a616f9d69 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:21.615905625 +0000 UTC m=+2.276448917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs") pod "network-metrics-daemon-mgb86" (UID: "ec058e17-0198-4c81-8ce4-2e3a616f9d69") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115905    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-registration-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115939    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-node-log\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.116932 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115952    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12b631c1-447d-43ee-9f5a-d77bfb04e03c-hosts-file\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115939    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-cnibin\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115981    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-cnibin\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115986    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-script-lib\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.115994    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-system-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116041    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czvds\" (UniqueName: \"kubernetes.io/projected/1fe78dfe-d1f0-4196-9832-702532afcaf8-kube-api-access-czvds\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116053    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-system-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-sys\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116086    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-system-cni-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116111    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtb7n\" (UniqueName: \"kubernetes.io/projected/3386381a-8b41-473b-9400-04d3d3eb2c7e-kube-api-access-mtb7n\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116134    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-sys-fs\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116147    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-sys\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116114    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-kubelet\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116182    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-system-cni-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116202    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-sys-fs\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116200    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.117693 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116240    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-kubelet\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116330    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116401    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-bin\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a065905-b86e-4326-a7d8-ec336cb938c7-ovn-node-metrics-cert\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e2f392-7651-4d4d-9c09-b4978aabcebf-host\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116479    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2efd8852-abb6-40d2-afda-40fe91bbd18f-agent-certs\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116506    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-bin\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-socket-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116521    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e2f392-7651-4d4d-9c09-b4978aabcebf-host\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-multus-certs\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-log-socket\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116602    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcqf\" (UniqueName: \"kubernetes.io/projected/ec058e17-0198-4c81-8ce4-2e3a616f9d69-kube-api-access-jkcqf\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116632    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-multus-certs\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrt52\" (UniqueName: \"kubernetes.io/projected/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kube-api-access-qrt52\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.118486 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116670    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysconfig\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116686    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-run\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-systemd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116735    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-run\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116748    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2efd8852-abb6-40d2-afda-40fe91bbd18f-konnectivity-ca\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-dbus\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmw9b\" (UniqueName: \"kubernetes.io/projected/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-kube-api-access-mmw9b\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3497b74a-87c3-4f27-91b7-dac04680a875-host-slash\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116648    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-socket-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116861    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-cnibin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116884    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-host\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116895    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysconfig\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116907    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-device-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-socket-dir-parent\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116950    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b631c1-447d-43ee-9f5a-d77bfb04e03c-tmp-dir\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116973    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzgw\" (UniqueName: \"kubernetes.io/projected/12b631c1-447d-43ee-9f5a-d77bfb04e03c-kube-api-access-mpzgw\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.119215 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116985    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-dbus\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116997    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-netd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117023    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-systemd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117023    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117033    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-host\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117057    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3497b74a-87c3-4f27-91b7-dac04680a875-iptables-alerter-script\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117083    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5tp\" (UniqueName: \"kubernetes.io/projected/3497b74a-87c3-4f27-91b7-dac04680a875-kube-api-access-6m5tp\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117106    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-bin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117127    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-etc-kubernetes\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117179    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-daemon-config\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117203    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-netns\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117229    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-hostroot\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117251    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-conf\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-systemd-units\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117315    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117389    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.119744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117443    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117473    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-cnibin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.116973    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-log-socket\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117536    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-netd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117595    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-etc-kubernetes\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117737    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-socket-dir-parent\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118023    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b631c1-447d-43ee-9f5a-d77bfb04e03c-tmp-dir\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118052    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118058    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2efd8852-abb6-40d2-afda-40fe91bbd18f-konnectivity-ca\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118100    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-device-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118148    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-hostroot\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118193    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-netns\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118297    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-conf\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118673    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-systemd-units\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118724    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-bin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.117268    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3497b74a-87c3-4f27-91b7-dac04680a875-host-slash\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.118920    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-tuned\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.119143    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-tmp\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.120181 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.119171    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3497b74a-87c3-4f27-91b7-dac04680a875-iptables-alerter-script\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.120667 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.119235    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-daemon-config\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.120667 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.119306    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a065905-b86e-4326-a7d8-ec336cb938c7-ovn-node-metrics-cert\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.120667 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.119405    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2efd8852-abb6-40d2-afda-40fe91bbd18f-agent-certs\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:21.171024 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.170921    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:21.171024 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.170956    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:21.171024 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.170970    2571 projected.go:196] Error preparing data for projected volume kube-api-access-8bbbk for pod openshift-network-diagnostics/network-check-target-ztv9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:21.171024 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.171030    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk podName:3ad94d5c-d364-4d68-8795-70af86c61c4e nodeName:}" failed. No retries permitted until 2026-10-31 18:20:21.671011472 +0000 UTC m=+2.331554783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8bbbk" (UniqueName: "kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk") pod "network-check-target-ztv9x" (UID: "3ad94d5c-d364-4d68-8795-70af86c61c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:21.174767 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.172109    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprjg\" (UniqueName: \"kubernetes.io/projected/0a065905-b86e-4326-a7d8-ec336cb938c7-kube-api-access-cprjg\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.174767 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.172815    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/a8e2f392-7651-4d4d-9c09-b4978aabcebf-kube-api-access-ldl6f\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.174767 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.172950    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtb7n\" (UniqueName: \"kubernetes.io/projected/3386381a-8b41-473b-9400-04d3d3eb2c7e-kube-api-access-mtb7n\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.174767 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.173121    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcqf\" (UniqueName: \"kubernetes.io/projected/ec058e17-0198-4c81-8ce4-2e3a616f9d69-kube-api-access-jkcqf\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:21.174767 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.173938    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czvds\" (UniqueName: \"kubernetes.io/projected/1fe78dfe-d1f0-4196-9832-702532afcaf8-kube-api-access-czvds\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.174767 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.174050    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzgw\" (UniqueName: \"kubernetes.io/projected/12b631c1-447d-43ee-9f5a-d77bfb04e03c-kube-api-access-mpzgw\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.174767 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.174122    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrt52\" (UniqueName: \"kubernetes.io/projected/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kube-api-access-qrt52\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.175172 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.174801    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5tp\" (UniqueName: \"kubernetes.io/projected/3497b74a-87c3-4f27-91b7-dac04680a875-kube-api-access-6m5tp\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.175260 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.175240    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmw9b\" (UniqueName: \"kubernetes.io/projected/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-kube-api-access-mmw9b\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.277455 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.277425    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc1d611432cbd40b2e3811e0cea4a6f.slice/crio-2105f6e721f592c0f6014006b097116a9857285c6406f46010196dd957d790f1 WatchSource:0}: Error finding container 2105f6e721f592c0f6014006b097116a9857285c6406f46010196dd957d790f1: Status 404 returned error can't find the container with id 2105f6e721f592c0f6014006b097116a9857285c6406f46010196dd957d790f1
Oct 31 18:20:21.278097 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.278075    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401f601c1d615f3dbfc663ac1fbee6e2.slice/crio-59110ac180d05d3383f18e51c3d2ef56424f0c974f43cd1b6e21bbcd044c6e28 WatchSource:0}: Error finding container 59110ac180d05d3383f18e51c3d2ef56424f0c974f43cd1b6e21bbcd044c6e28: Status 404 returned error can't find the container with id 59110ac180d05d3383f18e51c3d2ef56424f0c974f43cd1b6e21bbcd044c6e28
Oct 31 18:20:21.279202 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.279184    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:20:21.283711 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.283696    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:20:21.286536 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.286515    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b37353a_7cbf_478e_b7c5_83a71f6c1986.slice/crio-0097d97e1ef0c7adeaeb69eb26f8808d654557ad46b60bbb3bdf47bf65d84532 WatchSource:0}: Error finding container 0097d97e1ef0c7adeaeb69eb26f8808d654557ad46b60bbb3bdf47bf65d84532: Status 404 returned error can't find the container with id 0097d97e1ef0c7adeaeb69eb26f8808d654557ad46b60bbb3bdf47bf65d84532
Oct 31 18:20:21.304043 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.304016    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:20:21.311927 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.311900    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e2f392_7651_4d4d_9c09_b4978aabcebf.slice/crio-82cbbb05cb44ecbf8c2dc8e3117ed3bd8fca4cc4c15893a923eddacb4159dbb3 WatchSource:0}: Error finding container 82cbbb05cb44ecbf8c2dc8e3117ed3bd8fca4cc4c15893a923eddacb4159dbb3: Status 404 returned error can't find the container with id 82cbbb05cb44ecbf8c2dc8e3117ed3bd8fca4cc4c15893a923eddacb4159dbb3
Oct 31 18:20:21.326551 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.326530    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:20:21.333705 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.333683    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3386381a_8b41_473b_9400_04d3d3eb2c7e.slice/crio-3f5039aa0259af9d5c470852b5c56ce454f9100a23979e263cb35a0ba4ec0b8d WatchSource:0}: Error finding container 3f5039aa0259af9d5c470852b5c56ce454f9100a23979e263cb35a0ba4ec0b8d: Status 404 returned error can't find the container with id 3f5039aa0259af9d5c470852b5c56ce454f9100a23979e263cb35a0ba4ec0b8d
Oct 31 18:20:21.335648 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.335630    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bx85h"
Oct 31 18:20:21.339253 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.339235    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:20:21.342881 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.342861    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe78dfe_d1f0_4196_9832_702532afcaf8.slice/crio-4b7695f34b7a55d47f3bb55a9990a93090dfd794267f1557d28f2ff1aa26a225 WatchSource:0}: Error finding container 4b7695f34b7a55d47f3bb55a9990a93090dfd794267f1557d28f2ff1aa26a225: Status 404 returned error can't find the container with id 4b7695f34b7a55d47f3bb55a9990a93090dfd794267f1557d28f2ff1aa26a225
Oct 31 18:20:21.347544 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.347518    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b631c1_447d_43ee_9f5a_d77bfb04e03c.slice/crio-89b6ce0535b96eba824e61d9d6c38ee6d1969cbb71aed8545acdd294c558236d WatchSource:0}: Error finding container 89b6ce0535b96eba824e61d9d6c38ee6d1969cbb71aed8545acdd294c558236d: Status 404 returned error can't find the container with id 89b6ce0535b96eba824e61d9d6c38ee6d1969cbb71aed8545acdd294c558236d
Oct 31 18:20:21.355601 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.355568    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:20:21.362631 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.362608    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3497b74a_87c3_4f27_91b7_dac04680a875.slice/crio-018a1eaae75ee0d8c6ef2cd2fabc62a6e253fc6f8a0ac41eeafe39536af77525 WatchSource:0}: Error finding container 018a1eaae75ee0d8c6ef2cd2fabc62a6e253fc6f8a0ac41eeafe39536af77525: Status 404 returned error can't find the container with id 018a1eaae75ee0d8c6ef2cd2fabc62a6e253fc6f8a0ac41eeafe39536af77525
Oct 31 18:20:21.373223 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.373203    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:21.378086 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.378065    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:20:21.380450 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.380429    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efd8852_abb6_40d2_afda_40fe91bbd18f.slice/crio-eb570da9627300e528f6a81428e9d8b7a91eb8e7a2557f2b41f573c3ac8168d6 WatchSource:0}: Error finding container eb570da9627300e528f6a81428e9d8b7a91eb8e7a2557f2b41f573c3ac8168d6: Status 404 returned error can't find the container with id eb570da9627300e528f6a81428e9d8b7a91eb8e7a2557f2b41f573c3ac8168d6
Oct 31 18:20:21.384615 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.384594    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be4d751_a0c8_4350_ac3f_8e12d3eca25c.slice/crio-213ffb619b0ac643025a6df70541d151cfa68decbcfada4520dcf63b1239286b WatchSource:0}: Error finding container 213ffb619b0ac643025a6df70541d151cfa68decbcfada4520dcf63b1239286b: Status 404 returned error can't find the container with id 213ffb619b0ac643025a6df70541d151cfa68decbcfada4520dcf63b1239286b
Oct 31 18:20:21.387707 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:21.387687    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:21.394748 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:20:21.394727    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a065905_b86e_4326_a7d8_ec336cb938c7.slice/crio-ce0d6948e36ad007965b74f660246b4f836c2b0cd11ee3c75e7136b268d39bca WatchSource:0}: Error finding container ce0d6948e36ad007965b74f660246b4f836c2b0cd11ee3c75e7136b268d39bca: Status 404 returned error can't find the container with id ce0d6948e36ad007965b74f660246b4f836c2b0cd11ee3c75e7136b268d39bca
Oct 31 18:20:21.397095 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.397074    2571 kuberuntime_manager.go:1707] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-qtzdz_openshift-ovn-kubernetes(0a065905-b86e-4326-a7d8-ec336cb938c7): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 31 18:20:21.398985 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.398959    2571 kuberuntime_manager.go:1707] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-qtzdz_openshift-ovn-kubernetes(0a065905-b86e-4326-a7d8-ec336cb938c7): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 31 18:20:21.403752 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:21.403706    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" podUID="0a065905-b86e-4326-a7d8-ec336cb938c7"
Oct 31 18:20:23.095565 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.095485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:23.095565 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.095544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:23.095754 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.095621    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:23.095754 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.095620    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:23.095754 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.095667    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret podName:28594e99-2b75-44a6-8922-315e6b680700 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:24.095656074 +0000 UTC m=+3.282322157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret") pod "global-pull-secret-syncer-m56gf" (UID: "28594e99-2b75-44a6-8922-315e6b680700") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:23.095754 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.095681    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs podName:ec058e17-0198-4c81-8ce4-2e3a616f9d69 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:24.095674696 +0000 UTC m=+3.282340766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs") pod "network-metrics-daemon-mgb86" (UID: "ec058e17-0198-4c81-8ce4-2e3a616f9d69") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:23.196493 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.196442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:23.197240 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.196632    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:23.197240 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.196660    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:23.197240 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.196673    2571 projected.go:196] Error preparing data for projected volume kube-api-access-8bbbk for pod openshift-network-diagnostics/network-check-target-ztv9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:23.197240 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.196754    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk podName:3ad94d5c-d364-4d68-8795-70af86c61c4e nodeName:}" failed. No retries permitted until 2026-10-31 18:20:24.196736377 +0000 UTC m=+3.383402460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bbbk" (UniqueName: "kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk") pod "network-check-target-ztv9x" (UID: "3ad94d5c-d364-4d68-8795-70af86c61c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:23.266712 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.266665    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"ce0d6948e36ad007965b74f660246b4f836c2b0cd11ee3c75e7136b268d39bca"}
Oct 31 18:20:23.267806 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.267732    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4g2hb" event={"ID":"8be4d751-a0c8-4350-ac3f-8e12d3eca25c","Type":"ContainerStarted","Data":"213ffb619b0ac643025a6df70541d151cfa68decbcfada4520dcf63b1239286b"}
Oct 31 18:20:23.268841 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.268778    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hgxrq" event={"ID":"2efd8852-abb6-40d2-afda-40fe91bbd18f","Type":"ContainerStarted","Data":"eb570da9627300e528f6a81428e9d8b7a91eb8e7a2557f2b41f573c3ac8168d6"}
Oct 31 18:20:23.269883 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.269801    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-25sgx" event={"ID":"3497b74a-87c3-4f27-91b7-dac04680a875","Type":"ContainerStarted","Data":"018a1eaae75ee0d8c6ef2cd2fabc62a6e253fc6f8a0ac41eeafe39536af77525"}
Oct 31 18:20:23.271273 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.271256    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bx85h" event={"ID":"1fe78dfe-d1f0-4196-9832-702532afcaf8","Type":"ContainerStarted","Data":"4b7695f34b7a55d47f3bb55a9990a93090dfd794267f1557d28f2ff1aa26a225"}
Oct 31 18:20:23.272383 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.272358    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc" event={"ID":"8b37353a-7cbf-478e-b7c5-83a71f6c1986","Type":"ContainerStarted","Data":"0097d97e1ef0c7adeaeb69eb26f8808d654557ad46b60bbb3bdf47bf65d84532"}
Oct 31 18:20:23.273417 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.273399    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"adc1d611432cbd40b2e3811e0cea4a6f","Type":"ContainerStarted","Data":"2105f6e721f592c0f6014006b097116a9857285c6406f46010196dd957d790f1"}
Oct 31 18:20:23.274383 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.274364    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"401f601c1d615f3dbfc663ac1fbee6e2","Type":"ContainerStarted","Data":"59110ac180d05d3383f18e51c3d2ef56424f0c974f43cd1b6e21bbcd044c6e28"}
Oct 31 18:20:23.275420 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.275401    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xjpst" event={"ID":"12b631c1-447d-43ee-9f5a-d77bfb04e03c","Type":"ContainerStarted","Data":"89b6ce0535b96eba824e61d9d6c38ee6d1969cbb71aed8545acdd294c558236d"}
Oct 31 18:20:23.275677 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:23.275608    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" podUID="0a065905-b86e-4326-a7d8-ec336cb938c7"
Oct 31 18:20:23.276434 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.276417    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" event={"ID":"3386381a-8b41-473b-9400-04d3d3eb2c7e","Type":"ContainerStarted","Data":"3f5039aa0259af9d5c470852b5c56ce454f9100a23979e263cb35a0ba4ec0b8d"}
Oct 31 18:20:23.277324 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.277306    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rlwgx" event={"ID":"a8e2f392-7651-4d4d-9c09-b4978aabcebf","Type":"ContainerStarted","Data":"82cbbb05cb44ecbf8c2dc8e3117ed3bd8fca4cc4c15893a923eddacb4159dbb3"}
Oct 31 18:20:23.297488 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.297445    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:23.396827 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.396754    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:23.412005 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.411982    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:23.572379 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.572311    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:22 +0000 UTC" deadline="2028-07-20 17:46:24.944606576 +0000 UTC"
Oct 31 18:20:23.572379 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:23.572370    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15071h26m1.372240845s"
Oct 31 18:20:24.104640 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:24.104599    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:24.104827 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:24.104666    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:24.104827 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.104811    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:24.104935 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.104873    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs podName:ec058e17-0198-4c81-8ce4-2e3a616f9d69 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:26.104854956 +0000 UTC m=+5.291521032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs") pod "network-metrics-daemon-mgb86" (UID: "ec058e17-0198-4c81-8ce4-2e3a616f9d69") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:24.104935 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.104933    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:24.105045 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.104967    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret podName:28594e99-2b75-44a6-8922-315e6b680700 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:26.104954816 +0000 UTC m=+5.291620910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret") pod "global-pull-secret-syncer-m56gf" (UID: "28594e99-2b75-44a6-8922-315e6b680700") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:24.205824 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:24.205789    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:24.206303 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.206072    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:24.206303 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.206090    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:24.206303 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.206099    2571 projected.go:196] Error preparing data for projected volume kube-api-access-8bbbk for pod openshift-network-diagnostics/network-check-target-ztv9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:24.206303 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.206143    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk podName:3ad94d5c-d364-4d68-8795-70af86c61c4e nodeName:}" failed. No retries permitted until 2026-10-31 18:20:26.206131431 +0000 UTC m=+5.392797501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bbbk" (UniqueName: "kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk") pod "network-check-target-ztv9x" (UID: "3ad94d5c-d364-4d68-8795-70af86c61c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:24.263642 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:24.263608    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:24.263820 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.263739    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:24.263878 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:24.263827    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:24.263929 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.263912    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:24.263988 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:24.263976    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:24.264073 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:24.264054    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:24.573148 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:24.573092    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:22 +0000 UTC" deadline="2028-07-11 22:20:09.0136331 +0000 UTC"
Oct 31 18:20:24.573148 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:24.573149    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14859h59m44.440488184s"
Oct 31 18:20:26.123628 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:26.123144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:26.123628 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:26.123235    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:26.123628 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.123351    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:26.123628 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.123412    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs podName:ec058e17-0198-4c81-8ce4-2e3a616f9d69 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:30.123393241 +0000 UTC m=+9.310059329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs") pod "network-metrics-daemon-mgb86" (UID: "ec058e17-0198-4c81-8ce4-2e3a616f9d69") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:26.123628 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.123483    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:26.123628 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.123520    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret podName:28594e99-2b75-44a6-8922-315e6b680700 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:30.123508957 +0000 UTC m=+9.310175031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret") pod "global-pull-secret-syncer-m56gf" (UID: "28594e99-2b75-44a6-8922-315e6b680700") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:26.223964 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:26.223931    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:26.224149 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.224128    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:26.224210 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.224152    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:26.224210 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.224163    2571 projected.go:196] Error preparing data for projected volume kube-api-access-8bbbk for pod openshift-network-diagnostics/network-check-target-ztv9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:26.224317 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.224216    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk podName:3ad94d5c-d364-4d68-8795-70af86c61c4e nodeName:}" failed. No retries permitted until 2026-10-31 18:20:30.224202838 +0000 UTC m=+9.410868910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bbbk" (UniqueName: "kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk") pod "network-check-target-ztv9x" (UID: "3ad94d5c-d364-4d68-8795-70af86c61c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:26.262861 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:26.262821    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:26.263022 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.262957    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:26.263022 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:26.263018    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:26.263126 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.263103    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:26.263186 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:26.263148    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:26.263233 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:26.263211    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:27.304364 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:27.304292    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hgxrq" event={"ID":"2efd8852-abb6-40d2-afda-40fe91bbd18f","Type":"ContainerStarted","Data":"3b0dd9c6c65da8e562695e095db447a2bb34d6b850997a270f7d05e3f5986f2d"}
Oct 31 18:20:27.308471 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:27.308415    2571 generic.go:356] "Generic (PLEG): container finished" podID="adc1d611432cbd40b2e3811e0cea4a6f" containerID="b0f7a6b276247df54ed639c4d6df9fc9843e3cd68f7f41f3ec14379a7d29d169" exitCode=0
Oct 31 18:20:27.308631 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:27.308508    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"adc1d611432cbd40b2e3811e0cea4a6f","Type":"ContainerDied","Data":"b0f7a6b276247df54ed639c4d6df9fc9843e3cd68f7f41f3ec14379a7d29d169"}
Oct 31 18:20:27.312838 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:27.312812    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"401f601c1d615f3dbfc663ac1fbee6e2","Type":"ContainerStarted","Data":"526dbadcd0b4f7e718eabfbb46a8fc61a0a17cbba4ddfc804b7df0d015eff5e6"}
Oct 31 18:20:27.322412 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:27.322389    2571 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="b2c02bc3807c57f874c00d27f61048ea799e6742e181e61da6b922677de168e0" exitCode=0
Oct 31 18:20:27.322508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:27.322424    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" event={"ID":"3386381a-8b41-473b-9400-04d3d3eb2c7e","Type":"ContainerDied","Data":"b2c02bc3807c57f874c00d27f61048ea799e6742e181e61da6b922677de168e0"}
Oct 31 18:20:27.533021 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:27.532957    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hgxrq" podStartSLOduration=2.384023114 podStartE2EDuration="6.532939302s" totalImagesPullingTime="4.148916188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:27.418861507 +0000 UTC m=+6.605527600" watchObservedRunningTime="2026-10-31 18:20:27.532939302 +0000 UTC m=+6.719605397"
Oct 31 18:20:28.263600 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:28.263564    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:28.263600 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:28.263588    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:28.263844 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:28.263567    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:28.263844 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:28.263709    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:28.263844 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:28.263788    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:28.263844 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:28.263839    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:28.329361 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:28.329317    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"adc1d611432cbd40b2e3811e0cea4a6f","Type":"ContainerStarted","Data":"fe2294365af9d121bef3ddec65b8ea1018f93fdbd1145f1a87508dbcfad17737"}
Oct 31 18:20:28.332189 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:28.332157    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rlwgx" event={"ID":"a8e2f392-7651-4d4d-9c09-b4978aabcebf","Type":"ContainerStarted","Data":"9c6254219a73e3551eb25386d196a75187db4e90515f7ac85a2d7f73ab89beeb"}
Oct 31 18:20:28.442265 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:28.442206    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal" podStartSLOduration=7.4421883189999996 podStartE2EDuration="7.442188319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:27.764058611 +0000 UTC m=+6.950724718" watchObservedRunningTime="2026-10-31 18:20:28.442188319 +0000 UTC m=+7.628854413"
Oct 31 18:20:28.556451 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:28.556350    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal" podStartSLOduration=7.556328486 podStartE2EDuration="7.556328486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:28.443357901 +0000 UTC m=+7.630024015" watchObservedRunningTime="2026-10-31 18:20:28.556328486 +0000 UTC m=+7.742994581"
Oct 31 18:20:29.101668 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:29.101538    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:29.102372 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:29.102127    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:29.216836 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:29.216681    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rlwgx" podStartSLOduration=3.272614352 podStartE2EDuration="8.216662647s" totalImagesPullingTime="4.944048295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:28.557099841 +0000 UTC m=+7.743765934" watchObservedRunningTime="2026-10-31 18:20:29.216662647 +0000 UTC m=+8.403328741"
Oct 31 18:20:29.338987 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:29.338413    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc" event={"ID":"8b37353a-7cbf-478e-b7c5-83a71f6c1986","Type":"ContainerStarted","Data":"c99194f9ea5690fa5af85d4c6efde44a002f8e32517501503c5d5ad484e68de9"}
Oct 31 18:20:30.159087 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.158704    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:30.159087 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.158788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:30.159087 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.158911    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:30.159087 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.158964    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret podName:28594e99-2b75-44a6-8922-315e6b680700 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:38.158948142 +0000 UTC m=+17.345614217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret") pod "global-pull-secret-syncer-m56gf" (UID: "28594e99-2b75-44a6-8922-315e6b680700") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:30.159087 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.158997    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:30.159087 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.159058    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs podName:ec058e17-0198-4c81-8ce4-2e3a616f9d69 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:38.159040066 +0000 UTC m=+17.345706153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs") pod "network-metrics-daemon-mgb86" (UID: "ec058e17-0198-4c81-8ce4-2e3a616f9d69") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:30.259249 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.259222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:30.259390 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.259372    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:30.259478 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.259399    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:30.259478 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.259411    2571 projected.go:196] Error preparing data for projected volume kube-api-access-8bbbk for pod openshift-network-diagnostics/network-check-target-ztv9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:30.259478 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.259479    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk podName:3ad94d5c-d364-4d68-8795-70af86c61c4e nodeName:}" failed. No retries permitted until 2026-10-31 18:20:38.25944424 +0000 UTC m=+17.446110327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bbbk" (UniqueName: "kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk") pod "network-check-target-ztv9x" (UID: "3ad94d5c-d364-4d68-8795-70af86c61c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:30.262668 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.262640    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:30.262814 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.262764    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:30.262896 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.262843    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:30.262948 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.262926    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:30.262999 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.262974    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:30.263051 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:30.263034    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:30.277900 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.277870    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 18:20:30.342753 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.342653    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bx85h" event={"ID":"1fe78dfe-d1f0-4196-9832-702532afcaf8","Type":"ContainerStarted","Data":"b51fea85ce8f5b4656b2fb07d960fd7b7d8744c45225000ed24dc68dbca676f3"}
Oct 31 18:20:30.345332 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.345273    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc" event={"ID":"8b37353a-7cbf-478e-b7c5-83a71f6c1986","Type":"ContainerStarted","Data":"49b8fffc2ce58b592776c35b850cfa3b80751b75ed063d3302f1fb8d65ab198b"}
Oct 31 18:20:30.347108 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.347040    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 18:20:30.347108 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.347057    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xjpst" event={"ID":"12b631c1-447d-43ee-9f5a-d77bfb04e03c","Type":"ContainerStarted","Data":"3b4fd98df7aa208ba0389ed1a57877eee47bb4b61f4b1460a977adaff1614ef2"}
Oct 31 18:20:30.457560 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.457505    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bx85h" podStartSLOduration=2.17362987 podStartE2EDuration="9.457487115s" totalImagesPullingTime="7.283857245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:30.456835803 +0000 UTC m=+9.643501887" watchObservedRunningTime="2026-10-31 18:20:30.457487115 +0000 UTC m=+9.644153208"
Oct 31 18:20:30.570517 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:30.570471    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xjpst" podStartSLOduration=2.307753736 podStartE2EDuration="9.570441572s" totalImagesPullingTime="7.262687836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:30.569925468 +0000 UTC m=+9.756591563" watchObservedRunningTime="2026-10-31 18:20:30.570441572 +0000 UTC m=+9.757107663"
Oct 31 18:20:31.255206 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:31.255091    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T18:20:30.277896232Z","UUID":"2de7e627-6c63-4a8e-8fc6-f05ca2c53c04","Handler":null,"Name":"","Endpoint":""}
Oct 31 18:20:31.256872 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:31.256848    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 18:20:31.257002 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:31.256879    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 18:20:31.351633 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:31.351587    2571 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="baffe93be09bea7630e8fbbd7fc6016f1af5cda17c9f012213d0b7eabfcdb6bd" exitCode=0
Oct 31 18:20:31.351633 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:31.351627    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" event={"ID":"3386381a-8b41-473b-9400-04d3d3eb2c7e","Type":"ContainerDied","Data":"baffe93be09bea7630e8fbbd7fc6016f1af5cda17c9f012213d0b7eabfcdb6bd"}
Oct 31 18:20:32.263289 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.263077    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:32.263412 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.263098    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:32.263412 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:32.263317    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:32.263412 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.263131    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:32.263412 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:32.263397    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:32.263557 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:32.263473    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:32.356232 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.356192    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4g2hb" event={"ID":"8be4d751-a0c8-4350-ac3f-8e12d3eca25c","Type":"ContainerStarted","Data":"664a79ee2d17412989a76950caebdcf4971d360d44d792e4031db27d0940e818"}
Oct 31 18:20:32.358478 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.358421    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-25sgx" event={"ID":"3497b74a-87c3-4f27-91b7-dac04680a875","Type":"ContainerStarted","Data":"d62efb7a4e527901cb6a9a0390a9a99364f8027512e6e89ddbc287c6461c3361"}
Oct 31 18:20:32.362227 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.362195    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc" event={"ID":"8b37353a-7cbf-478e-b7c5-83a71f6c1986","Type":"ContainerStarted","Data":"be7251d802390bd6db4f83a9c81ae77650787d0b1ebc69dc876e2f10ef302cec"}
Oct 31 18:20:32.469923 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.469878    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4g2hb" podStartSLOduration=2.494946911 podStartE2EDuration="11.469866078s" totalImagesPullingTime="8.974919167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:32.469637518 +0000 UTC m=+11.656303609" watchObservedRunningTime="2026-10-31 18:20:32.469866078 +0000 UTC m=+11.656532168"
Oct 31 18:20:32.584443 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.584376    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc" podStartSLOduration=3.068087055 podStartE2EDuration="11.584364624s" totalImagesPullingTime="8.516277569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:32.584198311 +0000 UTC m=+11.770864403" watchObservedRunningTime="2026-10-31 18:20:32.584364624 +0000 UTC m=+11.771030716"
Oct 31 18:20:32.695252 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:32.695199    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-25sgx" podStartSLOduration=4.103819429 podStartE2EDuration="11.695181514s" totalImagesPullingTime="7.591362085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:32.695076541 +0000 UTC m=+11.881742612" watchObservedRunningTime="2026-10-31 18:20:32.695181514 +0000 UTC m=+11.881847606"
Oct 31 18:20:33.366401 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:33.366363    2571 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="3725d11fa5414dfc393507b2231ba4ad2a4592cb5f98fcf3590d721f44329391" exitCode=0
Oct 31 18:20:33.366893 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:33.366481    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" event={"ID":"3386381a-8b41-473b-9400-04d3d3eb2c7e","Type":"ContainerDied","Data":"3725d11fa5414dfc393507b2231ba4ad2a4592cb5f98fcf3590d721f44329391"}
Oct 31 18:20:34.263489 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:34.263437    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:34.263643 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:34.263496    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:34.263643 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:34.263574    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:34.263643 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:34.263630    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:34.263817 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:34.263778    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:34.263875 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:34.263859    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:35.373799 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:35.373598    2571 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="dd1574ca726778db56205c8ddf96ca4e9b65bf2e7aa7f26c966221853a1f2cc3" exitCode=0
Oct 31 18:20:35.373799 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:35.373683    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" event={"ID":"3386381a-8b41-473b-9400-04d3d3eb2c7e","Type":"ContainerDied","Data":"dd1574ca726778db56205c8ddf96ca4e9b65bf2e7aa7f26c966221853a1f2cc3"}
Oct 31 18:20:36.263161 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:36.263132    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:36.263308 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:36.263224    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:36.263308 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:36.263243    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:36.263410 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:36.263340    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:36.263472 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:36.263426    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:36.263556 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:36.263533    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:38.040651 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.040608    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:38.041044 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.040759    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 18:20:38.041397 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.041375    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:20:38.218127 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.218042    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:38.218127 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.218094    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:38.218345 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.218179    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:38.218345 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.218208    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:38.218345 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.218252    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret podName:28594e99-2b75-44a6-8922-315e6b680700 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.218230699 +0000 UTC m=+33.404896786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret") pod "global-pull-secret-syncer-m56gf" (UID: "28594e99-2b75-44a6-8922-315e6b680700") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:38.218345 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.218273    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs podName:ec058e17-0198-4c81-8ce4-2e3a616f9d69 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.218262154 +0000 UTC m=+33.404928227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs") pod "network-metrics-daemon-mgb86" (UID: "ec058e17-0198-4c81-8ce4-2e3a616f9d69") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:38.263386 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.263350    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:38.263574 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.263414    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:38.263574 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.263561    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:38.263661 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.263555    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:38.263661 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.263621    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:38.263747 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.263700    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:38.318939 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.318900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:38.319129 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.319046    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:38.319129 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.319060    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:38.319129 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.319070    2571 projected.go:196] Error preparing data for projected volume kube-api-access-8bbbk for pod openshift-network-diagnostics/network-check-target-ztv9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:38.319129 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:38.319129    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk podName:3ad94d5c-d364-4d68-8795-70af86c61c4e nodeName:}" failed. No retries permitted until 2026-10-31 18:20:54.319114677 +0000 UTC m=+33.505780762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bbbk" (UniqueName: "kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk") pod "network-check-target-ztv9x" (UID: "3ad94d5c-d364-4d68-8795-70af86c61c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:38.383911 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.383876    2571 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="da5e38825292c592f1a61ec65f6f43d3f1afa2dd99c37654cad4e6efb5ef2017" exitCode=0
Oct 31 18:20:38.384091 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:38.383907    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" event={"ID":"3386381a-8b41-473b-9400-04d3d3eb2c7e","Type":"ContainerDied","Data":"da5e38825292c592f1a61ec65f6f43d3f1afa2dd99c37654cad4e6efb5ef2017"}
Oct 31 18:20:39.389011 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:39.388976    2571 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="0ce66beea8f3997cd7f3db2e0b2d59534e1140835daecd790fcc5a4d508c1765" exitCode=0
Oct 31 18:20:39.389472 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:39.389012    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" event={"ID":"3386381a-8b41-473b-9400-04d3d3eb2c7e","Type":"ContainerDied","Data":"0ce66beea8f3997cd7f3db2e0b2d59534e1140835daecd790fcc5a4d508c1765"}
Oct 31 18:20:40.262867 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:40.262827    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:40.263038 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:40.262834    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:40.263038 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:40.262958    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:40.263038 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:40.263014    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:40.263038 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:40.262833    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:40.263165 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:40.263085    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:40.394105 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:40.394078    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" event={"ID":"3386381a-8b41-473b-9400-04d3d3eb2c7e","Type":"ContainerStarted","Data":"aac054d843096bca56575f47ce7c2d79ea87dcd2fba75ff322cec8595583e99e"}
Oct 31 18:20:40.613297 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:40.613190    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rg4dh" podStartSLOduration=8.257851161 podStartE2EDuration="19.613175912s" totalImagesPullingTime="11.355324751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:40.613088605 +0000 UTC m=+19.799754696" watchObservedRunningTime="2026-10-31 18:20:40.613175912 +0000 UTC m=+19.799842006"
Oct 31 18:20:42.262988 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:42.262958    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:42.263397 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:42.263000    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:42.263397 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:42.263060    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:42.263397 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:42.263098    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:42.263397 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:42.263161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:42.263397 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:42.263221    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:44.263450 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:44.263414    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:44.263450 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:44.263439    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:44.263450 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:44.263473    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:44.264309 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:44.263590    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:44.264309 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:44.263633    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:44.264309 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:44.263710    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:46.263270 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:46.263244    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:46.263732 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:46.263243    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:46.263732 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:46.263372    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:46.263732 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:46.263247    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:46.263732 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:46.263449    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:46.263732 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:46.263551    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:48.263602 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:48.263568    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:48.263602 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:48.263597    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:48.264165 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:48.263604    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:48.264165 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:48.263671    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:48.264165 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:48.263780    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:48.264165 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:48.263863    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:50.263283 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.263120    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:50.263659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.263191    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:50.263659 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:50.263362    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:50.263659 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.263201    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:50.263659 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:50.263471    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:50.263659 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:50.263509    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:50.417426 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.417389    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"3e251308f7cae36dd2ba4642643ff77894d53a3036215423269f9beca9edd192"}
Oct 31 18:20:50.417599 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.417490    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"58475e256d94670392ae0702defd7a3edac53e9fd1dddac800c67a99ef1e44a1"}
Oct 31 18:20:50.417599 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.417523    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"9ed4c1918d664ebb3a725b5d8b7a2035100e22b588a8977773793598a8f2e8c8"}
Oct 31 18:20:50.417599 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.417550    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"8ccb85210fd7c4b7a7e7bd483ff6db05a2b3016cabc7d39b08de9d717f63a487"}
Oct 31 18:20:50.417599 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.417576    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"db01ff18649ac53159bbf32cc1d8fb6b228158be4b101d36332a984c8a236b05"}
Oct 31 18:20:50.417740 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:50.417601    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"fae868e1a2c562b323d2caacfe8b943e4a64789fb25dc2e1785d4775dd645430"}
Oct 31 18:20:52.263623 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:52.263589    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:52.263623 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:52.263614    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:52.264099 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:52.263664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:52.264099 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:52.263752    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:52.264099 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:52.263864    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:52.264099 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:52.263942    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:53.426985 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:53.426955    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"3a2c6673dfe5fc2a66e5acfa8def4716cf8c808ea58cfe6bba0433d7b2599f3b"}
Oct 31 18:20:53.680820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:53.680757    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xjpst_12b631c1-447d-43ee-9f5a-d77bfb04e03c/dns-node-resolver/0.log"
Oct 31 18:20:54.229315 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.229288    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:54.229488 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.229323    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:54.229488 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.229426    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:54.229488 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.229430    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:54.229588 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.229508    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs podName:ec058e17-0198-4c81-8ce4-2e3a616f9d69 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:26.229477674 +0000 UTC m=+65.416143760 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs") pod "network-metrics-daemon-mgb86" (UID: "ec058e17-0198-4c81-8ce4-2e3a616f9d69") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:54.229588 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.229521    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret podName:28594e99-2b75-44a6-8922-315e6b680700 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:26.229515554 +0000 UTC m=+65.416181625 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret") pod "global-pull-secret-syncer-m56gf" (UID: "28594e99-2b75-44a6-8922-315e6b680700") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:54.263416 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.263395    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:54.263519 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.263398    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:54.263519 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.263486    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:54.263593 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.263398    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:54.263593 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.263572    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:54.263648 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.263613    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:54.330119 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.330097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:54.330225 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.330214    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:54.330268 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.330228    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:54.330268 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.330237    2571 projected.go:196] Error preparing data for projected volume kube-api-access-8bbbk for pod openshift-network-diagnostics/network-check-target-ztv9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:54.330326 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:54.330272    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk podName:3ad94d5c-d364-4d68-8795-70af86c61c4e nodeName:}" failed. No retries permitted until 2026-10-31 18:21:26.330261241 +0000 UTC m=+65.516927310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8bbbk" (UniqueName: "kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk") pod "network-check-target-ztv9x" (UID: "3ad94d5c-d364-4d68-8795-70af86c61c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:54.542469 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.542428    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hgxrq_2efd8852-abb6-40d2-afda-40fe91bbd18f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:54.720652 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.720624    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rlwgx_a8e2f392-7651-4d4d-9c09-b4978aabcebf/node-ca/0.log"
Oct 31 18:20:54.771496 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:54.771260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hgxrq_2efd8852-abb6-40d2-afda-40fe91bbd18f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:55.016263 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:55.016227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hgxrq_2efd8852-abb6-40d2-afda-40fe91bbd18f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:55.433410 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:55.433371    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" event={"ID":"0a065905-b86e-4326-a7d8-ec336cb938c7","Type":"ContainerStarted","Data":"19ec03e706e26d70904254eaf2ffbacc5991dd0137c6535b32872b97a925665b"}
Oct 31 18:20:55.433680 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:55.433663    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:55.433760 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:55.433736    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:55.449624 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:55.449602    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:55.707294 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:55.707214    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz" podStartSLOduration=7.493048423 podStartE2EDuration="34.707203609s" totalImagesPullingTime="27.214155186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:55.706969726 +0000 UTC m=+34.893635817" watchObservedRunningTime="2026-10-31 18:20:55.707203609 +0000 UTC m=+34.893869700"
Oct 31 18:20:56.263234 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:56.263204    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:56.263234 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:56.263221    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:56.263436 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:56.263326    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:56.263436 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:56.263366    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:56.263553 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:56.263446    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:56.263553 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:56.263527    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:20:56.435744 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:56.435721    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:56.450702 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:56.450675    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:20:58.263584 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:58.263550    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:20:58.263584 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:58.263574    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:20:58.264089 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:58.263689    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:20:58.264089 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:20:58.263723    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:20:58.264089 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:58.263797    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:20:58.264089 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:20:58.263857    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:00.262909 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:00.262878    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:00.263359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:00.262878    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:00.263359 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:00.262980    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:00.263359 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:00.262878    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:00.263359 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:00.263064    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:00.263359 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:00.263172    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:02.263389 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:02.263359    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:02.263820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:02.263452    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:02.263820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:02.263484    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:02.263820 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:02.263581    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:02.263820 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:02.263663    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:02.263820 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:02.263721    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:04.262788 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:04.262758    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:04.263165 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:04.262854    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:04.263165 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:04.262862    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:04.263165 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:04.262881    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:04.263165 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:04.262942    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:04.263165 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:04.263005    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:06.262697 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:06.262665    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:06.263075 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:06.262706    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:06.263075 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:06.262786    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:06.263075 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:06.262818    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:06.263075 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:06.262867    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:06.263075 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:06.262959    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:08.263047 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:08.263008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:08.263047 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:08.263032    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:08.263514 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:08.263141    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:08.263514 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:08.263185    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:08.263514 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:08.263254    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:08.263514 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:08.263304    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:10.263303 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:10.263273    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:10.263722 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:10.263372    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:10.263722 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:10.263387    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:10.263722 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:10.263443    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:10.263722 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:10.263472    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:10.263722 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:10.263566    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:12.262883 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.262810    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:12.263373 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.262853    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:12.263603 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:12.263582    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:12.263720 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:12.263409    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:12.263799 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.262866    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:12.263891 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:12.263874    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:12.331501 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.331449    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m56gf"]
Oct 31 18:21:12.335125 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.335093    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgb86"]
Oct 31 18:21:12.335249 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.335153    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ztv9x"]
Oct 31 18:21:12.469033 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.469000    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:12.469033 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.469014    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:12.469248 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:12.469096    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:12.469248 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:12.469102    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:12.469248 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:12.469177    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:12.469352 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:12.469247    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:14.262857 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:14.262828    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:14.263268 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:14.262913    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:14.263268 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:14.262922    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:14.263268 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:14.262992    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:14.263268 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:14.263005    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:14.263268 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:14.263087    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:16.262912 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:16.262879    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:16.263373 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:16.262973    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:16.263373 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:16.263076    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:16.263373 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:16.263093    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ztv9x" podUID="3ad94d5c-d364-4d68-8795-70af86c61c4e"
Oct 31 18:21:16.263373 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:16.263153    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m56gf" podUID="28594e99-2b75-44a6-8922-315e6b680700"
Oct 31 18:21:16.263373 ip-10-0-132-99 kubenswrapper[2571]: E1031 18:21:16.263215    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgb86" podUID="ec058e17-0198-4c81-8ce4-2e3a616f9d69"
Oct 31 18:21:17.143240 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.143212    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeReady"
Oct 31 18:21:17.143385 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.143313    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 18:21:17.296733 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.296701    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8z5vc"]
Oct 31 18:21:17.299115 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.299099    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g8rpt"]
Oct 31 18:21:17.299252 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.299236    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.301735 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.301712    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.342987 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.342968    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zkb25"]
Oct 31 18:21:17.345024 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.345003    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:17.350987 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.350965    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 18:21:17.351231 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.351218    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5t2xq\""
Oct 31 18:21:17.351292 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.351246    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 18:21:17.352831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.352817    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 18:21:17.353057 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.353041    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkjx2\""
Oct 31 18:21:17.353057 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.353054    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 18:21:17.356856 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.356837    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 18:21:17.356856 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.356846    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 18:21:17.392581 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.392551    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8z5vc"]
Oct 31 18:21:17.393128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.393110    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g8rpt"]
Oct 31 18:21:17.394976 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.394938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97e5d8ad-0765-4815-a495-53de9ca7b24b-crio-socket\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.395022 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.394981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrltf\" (UniqueName: \"kubernetes.io/projected/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-api-access-zrltf\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.395097 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.395065    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a446fa-5d09-4f26-9b5d-72f3fbe42064-tmp-dir\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.395097 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.395099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a446fa-5d09-4f26-9b5d-72f3fbe42064-config-volume\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.395516 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.395279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dchr\" (UniqueName: \"kubernetes.io/projected/69a446fa-5d09-4f26-9b5d-72f3fbe42064-kube-api-access-8dchr\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.395616 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.395559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.395689 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.395666    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97e5d8ad-0765-4815-a495-53de9ca7b24b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.395765 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.395748    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a446fa-5d09-4f26-9b5d-72f3fbe42064-metrics-tls\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.395819 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.395797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97e5d8ad-0765-4815-a495-53de9ca7b24b-data-volume\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.403879 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.403859    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zkb25"]
Oct 31 18:21:17.451594 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.451566    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 18:21:17.451702 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.451681    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms9mk\""
Oct 31 18:21:17.451763 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.451751    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 18:21:17.451806 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.451758    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 18:21:17.496221 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97e5d8ad-0765-4815-a495-53de9ca7b24b-crio-socket\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.496334 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496235    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrltf\" (UniqueName: \"kubernetes.io/projected/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-api-access-zrltf\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.496334 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-cert\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:17.496334 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496305    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64wql\" (UniqueName: \"kubernetes.io/projected/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-kube-api-access-64wql\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:17.496498 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496364    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a446fa-5d09-4f26-9b5d-72f3fbe42064-tmp-dir\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.496498 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a446fa-5d09-4f26-9b5d-72f3fbe42064-config-volume\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.496498 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496445    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dchr\" (UniqueName: \"kubernetes.io/projected/69a446fa-5d09-4f26-9b5d-72f3fbe42064-kube-api-access-8dchr\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.496498 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496446    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97e5d8ad-0765-4815-a495-53de9ca7b24b-crio-socket\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.496695 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.496695 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496685    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97e5d8ad-0765-4815-a495-53de9ca7b24b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.496795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496717    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a446fa-5d09-4f26-9b5d-72f3fbe42064-tmp-dir\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.496795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a446fa-5d09-4f26-9b5d-72f3fbe42064-metrics-tls\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.496795 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.496777    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97e5d8ad-0765-4815-a495-53de9ca7b24b-data-volume\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.497055 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.497036    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97e5d8ad-0765-4815-a495-53de9ca7b24b-data-volume\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.497180 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.497159    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a446fa-5d09-4f26-9b5d-72f3fbe42064-config-volume\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.497216 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.497194    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.500677 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.500657    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a446fa-5d09-4f26-9b5d-72f3fbe42064-metrics-tls\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.500677 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.500671    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97e5d8ad-0765-4815-a495-53de9ca7b24b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.552134 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.552115    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dchr\" (UniqueName: \"kubernetes.io/projected/69a446fa-5d09-4f26-9b5d-72f3fbe42064-kube-api-access-8dchr\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.552208 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.552148    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrltf\" (UniqueName: \"kubernetes.io/projected/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-api-access-zrltf\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.598020 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.597983    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-cert\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:17.598020 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.598018    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64wql\" (UniqueName: \"kubernetes.io/projected/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-kube-api-access-64wql\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:17.600634 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.600595    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-cert\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:17.612629 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.612578    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:17.620870 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.620834    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:17.654614 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.654592    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64wql\" (UniqueName: \"kubernetes.io/projected/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-kube-api-access-64wql\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:17.954439 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:17.954374    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:18.105569 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.105543    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8z5vc"]
Oct 31 18:21:18.108922 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.108615    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g8rpt"]
Oct 31 18:21:18.110143 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:18.110122    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a446fa_5d09_4f26_9b5d_72f3fbe42064.slice/crio-a00ba90066290cc95585207e999b86e0b3da4b2b3aa11f4afd5dce756983b2f7 WatchSource:0}: Error finding container a00ba90066290cc95585207e999b86e0b3da4b2b3aa11f4afd5dce756983b2f7: Status 404 returned error can't find the container with id a00ba90066290cc95585207e999b86e0b3da4b2b3aa11f4afd5dce756983b2f7
Oct 31 18:21:18.112200 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:18.112179    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e5d8ad_0765_4815_a495_53de9ca7b24b.slice/crio-8df7b013ba992048504b55b5954e8649371266a700cbc03b849d4b0604a8d704 WatchSource:0}: Error finding container 8df7b013ba992048504b55b5954e8649371266a700cbc03b849d4b0604a8d704: Status 404 returned error can't find the container with id 8df7b013ba992048504b55b5954e8649371266a700cbc03b849d4b0604a8d704
Oct 31 18:21:18.262847 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.262668    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:18.262955 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.262747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:18.263007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.262755    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:18.315286 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.315256    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 18:21:18.315830 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.315292    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vv64h\""
Oct 31 18:21:18.315830 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.315308    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 18:21:18.315830 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.315486    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 18:21:18.315830 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.315488    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-77xvx\""
Oct 31 18:21:18.315830 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.315533    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7mm2n\""
Oct 31 18:21:18.315830 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.315553    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 18:21:18.329970 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.329949    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zkb25"]
Oct 31 18:21:18.333379 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:18.333360    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f40051_3f1d_4d4f_9b06_b5f2cf7f5882.slice/crio-ea3ab49407dac02aad13cc91281751709100163ad3a967086ca9df9775e0b579 WatchSource:0}: Error finding container ea3ab49407dac02aad13cc91281751709100163ad3a967086ca9df9775e0b579: Status 404 returned error can't find the container with id ea3ab49407dac02aad13cc91281751709100163ad3a967086ca9df9775e0b579
Oct 31 18:21:18.481947 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.481862    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zkb25" event={"ID":"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882","Type":"ContainerStarted","Data":"ea3ab49407dac02aad13cc91281751709100163ad3a967086ca9df9775e0b579"}
Oct 31 18:21:18.483201 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.483173    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8rpt" event={"ID":"97e5d8ad-0765-4815-a495-53de9ca7b24b","Type":"ContainerStarted","Data":"24717f486c87f80d06cd001f704d4812d491d7395c6201c143d9d2c38eb314f0"}
Oct 31 18:21:18.483307 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.483223    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8rpt" event={"ID":"97e5d8ad-0765-4815-a495-53de9ca7b24b","Type":"ContainerStarted","Data":"8df7b013ba992048504b55b5954e8649371266a700cbc03b849d4b0604a8d704"}
Oct 31 18:21:18.484182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:18.484164    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8z5vc" event={"ID":"69a446fa-5d09-4f26-9b5d-72f3fbe42064","Type":"ContainerStarted","Data":"a00ba90066290cc95585207e999b86e0b3da4b2b3aa11f4afd5dce756983b2f7"}
Oct 31 18:21:20.492354 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:20.492316    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8rpt" event={"ID":"97e5d8ad-0765-4815-a495-53de9ca7b24b","Type":"ContainerStarted","Data":"b2aa08f5b54cfcdff9c63f3aaff07822a83dd2d097c8c7d7a433a39287cc6a48"}
Oct 31 18:21:21.097914 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.097880    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-8cp96"]
Oct 31 18:21:21.119812 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.119784    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:21.172382 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.172322    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 18:21:21.172958 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.172937    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 18:21:21.173093 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.173071    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-b8zc6\""
Oct 31 18:21:21.222690 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.222666    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-8cp96"]
Oct 31 18:21:21.225129 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.225110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpss5\" (UniqueName: \"kubernetes.io/projected/deca97c4-946e-4974-b30e-dd0595a94633-kube-api-access-cpss5\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:21.225250 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.225152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deca97c4-946e-4974-b30e-dd0595a94633-tmp\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:21.251558 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.251288    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"]
Oct 31 18:21:21.267524 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.266435    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.314417 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.314392    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"]
Oct 31 18:21:21.322372 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.322352    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 18:21:21.322517 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.322444    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 18:21:21.322573 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.322550    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 18:21:21.322646 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.322628    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bm89f\""
Oct 31 18:21:21.322767 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.322749    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 18:21:21.322857 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.322839    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 18:21:21.325805 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.325778    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deca97c4-946e-4974-b30e-dd0595a94633-tmp\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:21.325895 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.325853    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpss5\" (UniqueName: \"kubernetes.io/projected/deca97c4-946e-4974-b30e-dd0595a94633-kube-api-access-cpss5\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:21.326245 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.326223    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deca97c4-946e-4974-b30e-dd0595a94633-tmp\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:21.387333 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.387303    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpss5\" (UniqueName: \"kubernetes.io/projected/deca97c4-946e-4974-b30e-dd0595a94633-kube-api-access-cpss5\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:21.427227 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.427151    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62m6\" (UniqueName: \"kubernetes.io/projected/461b92ce-a67b-4dbe-924b-9e7c84f07a02-kube-api-access-v62m6\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.427227 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.427206    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.427385 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.427255    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/461b92ce-a67b-4dbe-924b-9e7c84f07a02-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.427385 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.427324    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.443904 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.443877    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:21.498095 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.497636    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8z5vc" event={"ID":"69a446fa-5d09-4f26-9b5d-72f3fbe42064","Type":"ContainerStarted","Data":"68bcd060c50366ba28f309fda733b7991afbb0d0de41266de495473c2d6697c8"}
Oct 31 18:21:21.498095 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.497731    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8z5vc" event={"ID":"69a446fa-5d09-4f26-9b5d-72f3fbe42064","Type":"ContainerStarted","Data":"e196d94736a2000b5a4807ef5b60e2b46f9823994b6c51ad517fc21237bbca28"}
Oct 31 18:21:21.498095 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.497841    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:21.528711 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.528654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62m6\" (UniqueName: \"kubernetes.io/projected/461b92ce-a67b-4dbe-924b-9e7c84f07a02-kube-api-access-v62m6\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.528829 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.528730    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.528829 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.528769    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/461b92ce-a67b-4dbe-924b-9e7c84f07a02-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.528935 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.528846    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.530428 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.530394    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/461b92ce-a67b-4dbe-924b-9e7c84f07a02-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.532151 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.532125    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.532420 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.532401    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.593732 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.593691    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62m6\" (UniqueName: \"kubernetes.io/projected/461b92ce-a67b-4dbe-924b-9e7c84f07a02-kube-api-access-v62m6\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.609745 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.609551    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:21.618032 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.617987    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-8z5vc" podStartSLOduration=2.060572563 podStartE2EDuration="4.617974115s" totalImagesPullingTime="2.557401552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:21.616066942 +0000 UTC m=+60.802733046" watchObservedRunningTime="2026-10-31 18:21:21.617974115 +0000 UTC m=+60.804640207"
Oct 31 18:21:21.891642 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:21.891613    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-8cp96"]
Oct 31 18:21:21.894691 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:21.894655    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeca97c4_946e_4974_b30e_dd0595a94633.slice/crio-cee41971d4a8889f0ee9c6c672b2e0710c00d4c16c44b98d31ff9d3deeac4cb9 WatchSource:0}: Error finding container cee41971d4a8889f0ee9c6c672b2e0710c00d4c16c44b98d31ff9d3deeac4cb9: Status 404 returned error can't find the container with id cee41971d4a8889f0ee9c6c672b2e0710c00d4c16c44b98d31ff9d3deeac4cb9
Oct 31 18:21:22.019452 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:22.019425    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"]
Oct 31 18:21:22.022539 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:22.022518    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461b92ce_a67b_4dbe_924b_9e7c84f07a02.slice/crio-c3a917921a0754afb3d482b95d56520d071e371af584de5cc2278953ca1e216a WatchSource:0}: Error finding container c3a917921a0754afb3d482b95d56520d071e371af584de5cc2278953ca1e216a: Status 404 returned error can't find the container with id c3a917921a0754afb3d482b95d56520d071e371af584de5cc2278953ca1e216a
Oct 31 18:21:22.501506 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:22.501448    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n" event={"ID":"461b92ce-a67b-4dbe-924b-9e7c84f07a02","Type":"ContainerStarted","Data":"c3a917921a0754afb3d482b95d56520d071e371af584de5cc2278953ca1e216a"}
Oct 31 18:21:22.502677 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:22.502653    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-8cp96" event={"ID":"deca97c4-946e-4974-b30e-dd0595a94633","Type":"ContainerStarted","Data":"cee41971d4a8889f0ee9c6c672b2e0710c00d4c16c44b98d31ff9d3deeac4cb9"}
Oct 31 18:21:22.505018 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:22.504993    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8rpt" event={"ID":"97e5d8ad-0765-4815-a495-53de9ca7b24b","Type":"ContainerStarted","Data":"843bce086c892528eacfbabe78e7acdf69593e93b5f3a83b62af34a2089b85e2"}
Oct 31 18:21:22.506470 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:22.506439    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zkb25" event={"ID":"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882","Type":"ContainerStarted","Data":"9f261fc025fc7d16a8612f1cb342548f3bc028b7e4535f8bbcb4da40e9b871c5"}
Oct 31 18:21:22.620300 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:22.620190    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g8rpt" podStartSLOduration=1.542630081 podStartE2EDuration="5.620176611s" totalImagesPullingTime="4.07754653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:21:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:22.619722872 +0000 UTC m=+61.806388983" watchObservedRunningTime="2026-10-31 18:21:22.620176611 +0000 UTC m=+61.806842703"
Oct 31 18:21:22.732018 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:22.731966    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zkb25" podStartSLOduration=1.676435067 podStartE2EDuration="5.731953111s" totalImagesPullingTime="4.055518044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:22.731030632 +0000 UTC m=+61.917696723" watchObservedRunningTime="2026-10-31 18:21:22.731953111 +0000 UTC m=+61.918619203"
Oct 31 18:21:24.516843 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:24.516804    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n" event={"ID":"461b92ce-a67b-4dbe-924b-9e7c84f07a02","Type":"ContainerStarted","Data":"792388c4f75e674ed6f8072df0e2cb8e98b572a6a1640453f08ad51acc623181"}
Oct 31 18:21:24.517285 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:24.516875    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n" event={"ID":"461b92ce-a67b-4dbe-924b-9e7c84f07a02","Type":"ContainerStarted","Data":"536f0a05e516133716c8caeb17d4aec0f729d349118395db7b445b3af442c72e"}
Oct 31 18:21:24.632763 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:24.632710    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n" podStartSLOduration=1.952304491 podStartE2EDuration="3.632696902s" totalImagesPullingTime="1.680392411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:24.631024898 +0000 UTC m=+63.817691009" watchObservedRunningTime="2026-10-31 18:21:24.632696902 +0000 UTC m=+63.819362990"
Oct 31 18:21:26.267050 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.267006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:26.267527 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.267064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:26.318241 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.318198    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 18:21:26.329861 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.329834    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:26.367541 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.367509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:26.367699 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.367624    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 18:21:26.380083 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.380059    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:26.418739 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.418715    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 18:21:26.443960 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.443925    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vv64h\""
Oct 31 18:21:26.454102 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.454080    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:26.478800 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.478781    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 18:21:26.491505 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.491482    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:26.515597 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.515578    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-77xvx\""
Oct 31 18:21:26.526366 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.526309    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:26.768830 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.768801    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7mm2n\""
Oct 31 18:21:26.778155 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.778089    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:26.834133 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.834027    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m56gf"]
Oct 31 18:21:26.841599 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:26.838912    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28594e99_2b75_44a6_8922_315e6b680700.slice/crio-fd649c93eab2f98b3997f01c56d2ff0f5c5385a2f6e0066591c4387d7dff8d5b WatchSource:0}: Error finding container fd649c93eab2f98b3997f01c56d2ff0f5c5385a2f6e0066591c4387d7dff8d5b: Status 404 returned error can't find the container with id fd649c93eab2f98b3997f01c56d2ff0f5c5385a2f6e0066591c4387d7dff8d5b
Oct 31 18:21:26.943727 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:26.943696    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgb86"]
Oct 31 18:21:26.947593 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:26.947543    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec058e17_0198_4c81_8ce4_2e3a616f9d69.slice/crio-ac28dc82845f0baae359492484e9bb817e94a43c75317221fe99aa941d1c0a11 WatchSource:0}: Error finding container ac28dc82845f0baae359492484e9bb817e94a43c75317221fe99aa941d1c0a11: Status 404 returned error can't find the container with id ac28dc82845f0baae359492484e9bb817e94a43c75317221fe99aa941d1c0a11
Oct 31 18:21:27.166344 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.166312    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ztv9x"]
Oct 31 18:21:27.169914 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:27.169882    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad94d5c_d364_4d68_8795_70af86c61c4e.slice/crio-093b014626279e61080b83681b46d60c99de3ef65c3f45de8aad99340e5d5c23 WatchSource:0}: Error finding container 093b014626279e61080b83681b46d60c99de3ef65c3f45de8aad99340e5d5c23: Status 404 returned error can't find the container with id 093b014626279e61080b83681b46d60c99de3ef65c3f45de8aad99340e5d5c23
Oct 31 18:21:27.530971 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.530894    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgb86" event={"ID":"ec058e17-0198-4c81-8ce4-2e3a616f9d69","Type":"ContainerStarted","Data":"ac28dc82845f0baae359492484e9bb817e94a43c75317221fe99aa941d1c0a11"}
Oct 31 18:21:27.532213 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.532179    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m56gf" event={"ID":"28594e99-2b75-44a6-8922-315e6b680700","Type":"ContainerStarted","Data":"fd649c93eab2f98b3997f01c56d2ff0f5c5385a2f6e0066591c4387d7dff8d5b"}
Oct 31 18:21:27.533707 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.533675    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ztv9x" event={"ID":"3ad94d5c-d364-4d68-8795-70af86c61c4e","Type":"ContainerStarted","Data":"093b014626279e61080b83681b46d60c99de3ef65c3f45de8aad99340e5d5c23"}
Oct 31 18:21:27.788873 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.788797    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"]
Oct 31 18:21:27.791494 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.791473    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.844095 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.844060    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 18:21:27.844248 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.844236    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dj9hm\""
Oct 31 18:21:27.879307 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.879275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.879443 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.879323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.879443 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.879355    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.879443 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.879415    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdx5n\" (UniqueName: \"kubernetes.io/projected/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-kube-api-access-sdx5n\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.894482 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.894431    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 18:21:27.895536 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.895510    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"]
Oct 31 18:21:27.965640 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.965609    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"]
Oct 31 18:21:27.967773 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.967749    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:27.975004 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.973794    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fgkm2"]
Oct 31 18:21:27.976137 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.976115    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.980503 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980390    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-wtmp\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.980503 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980431    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.980503 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:27.980712 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx5n\" (UniqueName: \"kubernetes.io/projected/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-kube-api-access-sdx5n\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.980712 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:27.980712 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.980884 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.980965 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980908    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-textfile\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.980965 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980944    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:27.981061 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.980974    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6bg2\" (UniqueName: \"kubernetes.io/projected/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-api-access-v6bg2\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:27.981061 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.981004    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-sys\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.981061 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.981027    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:27.988918 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.988094    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.988918 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.988165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-root\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.988918 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.988216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.988918 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.988260    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.988918 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.988298    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzp8k\" (UniqueName: \"kubernetes.io/projected/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-kube-api-access-jzp8k\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.988918 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.988330    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:27.988918 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.988370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-tls\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.988918 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.988407    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-metrics-client-ca\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:27.996901 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.993161    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:27.999615 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:27.999543    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:28.019956 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.019931    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 18:21:28.019956 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.019949    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qqqsk\""
Oct 31 18:21:28.020111 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.020069    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 18:21:28.020202 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.020184    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 18:21:28.058404 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.058325    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"]
Oct 31 18:21:28.067904 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.067841    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx5n\" (UniqueName: \"kubernetes.io/projected/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-kube-api-access-sdx5n\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:28.089620 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089592    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzp8k\" (UniqueName: \"kubernetes.io/projected/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-kube-api-access-jzp8k\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.089769 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089637    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.089769 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-tls\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.089769 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-metrics-client-ca\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.089769 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089718    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-wtmp\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.089769 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.090009 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.090009 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.090009 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089843    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-textfile\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.090009 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089877    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.090009 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6bg2\" (UniqueName: \"kubernetes.io/projected/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-api-access-v6bg2\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.090009 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-sys\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.090009 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.090009 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.089987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-root\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.090356 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.090016    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.091662 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.090746    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-textfile\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.091662 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.090757    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.091662 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.090803    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-sys\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.091662 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.091422    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.091662 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.091583    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-wtmp\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.091662 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.091621    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-metrics-client-ca\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.091662 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.091635    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-root\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.092485 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.092427    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.093981 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.093935    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.094057 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.093982    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.106626 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.106606    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:28.118395 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.118375    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 18:21:28.122249 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.122224    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.146962 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.146939    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 18:21:28.148141 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.147769    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 18:21:28.148141 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.147860    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9l6d9\""
Oct 31 18:21:28.150257 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.150232    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-tls\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.151224 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.151179    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzp8k\" (UniqueName: \"kubernetes.io/projected/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-kube-api-access-jzp8k\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.153473 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.153433    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.154977 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.154921    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6bg2\" (UniqueName: \"kubernetes.io/projected/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-api-access-v6bg2\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.303540 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.303506    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:28.317882 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.317857    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:28.459935 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.459898    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:28.541024 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.540989    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgkm2" event={"ID":"83fe51c6-ee5f-4ccf-a934-6d42a27d475b","Type":"ContainerStarted","Data":"1e99dc45621c676dae190438818cba7c89449d024ff30eb3ba5e322562f8399d"}
Oct 31 18:21:28.543350 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.543165    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"]
Oct 31 18:21:28.547032 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:28.547005    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445c9865_e1dc_467f_bb6f_f29e8c6b7f68.slice/crio-aacf2191f4fd9790f551b62643322993358b8dd5805ed54198a9dda1ec8bc97d WatchSource:0}: Error finding container aacf2191f4fd9790f551b62643322993358b8dd5805ed54198a9dda1ec8bc97d: Status 404 returned error can't find the container with id aacf2191f4fd9790f551b62643322993358b8dd5805ed54198a9dda1ec8bc97d
Oct 31 18:21:28.707270 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:28.707242    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"]
Oct 31 18:21:28.713119 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:28.712994    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c82ec5_b112_4e22_9e89_25b3a69fe6a2.slice/crio-3dac68756a73a04f709f12b3d6c1eefa758c545abf17718f7b5a361f7e6cf4f4 WatchSource:0}: Error finding container 3dac68756a73a04f709f12b3d6c1eefa758c545abf17718f7b5a361f7e6cf4f4: Status 404 returned error can't find the container with id 3dac68756a73a04f709f12b3d6c1eefa758c545abf17718f7b5a361f7e6cf4f4
Oct 31 18:21:29.192305 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.192269    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7986dc697c-f54gg"]
Oct 31 18:21:29.194487 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.194446    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.247368 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.247326    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 18:21:29.247528 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.247431    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 18:21:29.247528 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.247513    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 18:21:29.247699 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.247608    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 18:21:29.248115 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.248096    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 18:21:29.248217 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.248203    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bcwr9\""
Oct 31 18:21:29.256393 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.256361    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7986dc697c-f54gg"]
Oct 31 18:21:29.301509 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.301477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.301635 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.301522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.301635 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.301547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.301635 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.301581    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.301729 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.301643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7jq\" (UniqueName: \"kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.301729 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.301683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.301729 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.301709    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.403154 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.403105    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7jq\" (UniqueName: \"kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.403337 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.403166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.403337 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.403194    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.403337 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.403221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.403337 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.403248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.403337 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.403275    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.403337 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.403309    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.403963 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.403903    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.404402 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.404149    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.404517 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.404483    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.404768 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.404712    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.407137 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.407061    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.407137 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.407127    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.461108 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.461027    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7jq\" (UniqueName: \"kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.508003 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.507970    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:29.551383 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.551288    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5" event={"ID":"65c82ec5-b112-4e22-9e89-25b3a69fe6a2","Type":"ContainerStarted","Data":"3dac68756a73a04f709f12b3d6c1eefa758c545abf17718f7b5a361f7e6cf4f4"}
Oct 31 18:21:29.556116 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.555972    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv" event={"ID":"445c9865-e1dc-467f-bb6f-f29e8c6b7f68","Type":"ContainerStarted","Data":"d3bf7aae2fc6fa3e4f6dc6458c97dd86019fa35bdf5608e9784463b77736ae89"}
Oct 31 18:21:29.556116 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.556032    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv" event={"ID":"445c9865-e1dc-467f-bb6f-f29e8c6b7f68","Type":"ContainerStarted","Data":"5213f651efebb8ec2ecd5624646d439c2a5839ed75aa5700c35eae4cd6fce97c"}
Oct 31 18:21:29.556116 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.556061    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv" event={"ID":"445c9865-e1dc-467f-bb6f-f29e8c6b7f68","Type":"ContainerStarted","Data":"aacf2191f4fd9790f551b62643322993358b8dd5805ed54198a9dda1ec8bc97d"}
Oct 31 18:21:29.559352 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.558953    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgb86" event={"ID":"ec058e17-0198-4c81-8ce4-2e3a616f9d69","Type":"ContainerStarted","Data":"9fcf329d40213372c545d89bc1782dce2c0363c51d4865d9c5239d304dc60311"}
Oct 31 18:21:29.559352 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.558999    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgb86" event={"ID":"ec058e17-0198-4c81-8ce4-2e3a616f9d69","Type":"ContainerStarted","Data":"f7678d243efe852f9392c9c3e9bfcfd1bec6cdcdf78325f10f0e6caa362198d9"}
Oct 31 18:21:29.674859 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.674793    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mgb86" podStartSLOduration=66.766696637 podStartE2EDuration="1m8.674774173s" totalImagesPullingTime="1.908077536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:29.673373467 +0000 UTC m=+68.860039553" watchObservedRunningTime="2026-10-31 18:21:29.674774173 +0000 UTC m=+68.861440269"
Oct 31 18:21:29.959067 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:29.959029    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7986dc697c-f54gg"]
Oct 31 18:21:29.968878 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:29.968843    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519485a6_66ad_46f3_b5bc_76e5bc68315e.slice/crio-d24d184aa2ec9bcac439f2d949c9945598897bc7a9cfdaa271a427f0f703e74c WatchSource:0}: Error finding container d24d184aa2ec9bcac439f2d949c9945598897bc7a9cfdaa271a427f0f703e74c: Status 404 returned error can't find the container with id d24d184aa2ec9bcac439f2d949c9945598897bc7a9cfdaa271a427f0f703e74c
Oct 31 18:21:30.149112 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.149077    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:21:30.153224 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.153179    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211257 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211177    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk88q\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211257 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211232    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211267    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211302    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211355    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211412    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211437    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211502    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211873 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211873 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211700    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.211873 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.211779    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.257643 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.257612    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 18:21:30.257813 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.257778    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 18:21:30.258745 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.258720    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 18:21:30.258871 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.258821    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l6bdm\""
Oct 31 18:21:30.258925 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.258893    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 18:21:30.259007 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.258993    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 18:21:30.259073 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.259060    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 18:21:30.259147 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.259134    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 18:21:30.259231 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.259215    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 18:21:30.259354 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.259339    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 18:21:30.264646 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.261948    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:21:30.312353 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk88q\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312535 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312369    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312535 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312403    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312535 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312535 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312535 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312535 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312535    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312560    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312625    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312656    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312689    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.312820 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.312731    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.313289 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.313106    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.314296 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.314267    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.315890 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.315861    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.319983 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.319937    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.321294 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.321158    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.322069 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.321976    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.322069 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.322001    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.323089 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.323048    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.324093 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.324056    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.324500 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.324448    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.325632 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.325611    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.326024 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.326004    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.376289 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.376214    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk88q\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.475786 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.475579    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:30.500496 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.498922    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"]
Oct 31 18:21:30.503995 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.503185    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.572846 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.572803    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgkm2" event={"ID":"83fe51c6-ee5f-4ccf-a934-6d42a27d475b","Type":"ContainerStarted","Data":"5c52af85dabc9e2725ce1a088597e26758994da56c1eed5cdf4f53438555c300"}
Oct 31 18:21:30.587638 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.587564    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7986dc697c-f54gg" event={"ID":"519485a6-66ad-46f3-b5bc-76e5bc68315e","Type":"ContainerStarted","Data":"d24d184aa2ec9bcac439f2d949c9945598897bc7a9cfdaa271a427f0f703e74c"}
Oct 31 18:21:30.601390 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.601360    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"]
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-tmp\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619472    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whmgd\" (UniqueName: \"kubernetes.io/projected/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-kube-api-access-whmgd\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-metrics-client-ca\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-grpc-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.620831 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.619581    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.635809 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.635780    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-97cx7\""
Oct 31 18:21:30.636638 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.636014    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 18:21:30.636638 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.636307    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 18:21:30.636638 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.636425    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fo5e1905l2a1u\""
Oct 31 18:21:30.636638 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.636514    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 18:21:30.636638 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.636550    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 18:21:30.636638 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.636587    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-tmp\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720387    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmgd\" (UniqueName: \"kubernetes.io/projected/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-kube-api-access-whmgd\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-metrics-client-ca\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720470    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720507    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-grpc-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.720608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.721128 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.721042    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-tmp\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.723180 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.722940    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-metrics-client-ca\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.724646 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.724620    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.727520 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.727358    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.727520 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.727404    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.727864 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.727828    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.729669 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.729645    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-grpc-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.729962 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.729931    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.784434 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.784378    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmgd\" (UniqueName: \"kubernetes.io/projected/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-kube-api-access-whmgd\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.830601 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.830343    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:30.946261 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:30.946152    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:21:30.950234 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:30.950194    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c130b20_c7b5_46a5_8c19_59ca36d172c8.slice/crio-8b7fff8965badd296b83c5c42e37cd3cbc19e1263f5aeb185d9c54612146838a WatchSource:0}: Error finding container 8b7fff8965badd296b83c5c42e37cd3cbc19e1263f5aeb185d9c54612146838a: Status 404 returned error can't find the container with id 8b7fff8965badd296b83c5c42e37cd3cbc19e1263f5aeb185d9c54612146838a
Oct 31 18:21:31.249887 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:31.249738    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"]
Oct 31 18:21:31.256581 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:31.255718    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f7cf5d_858f_4ae2_b53e_c64881398c3f.slice/crio-59805563b901f5adace385924880f9fa196174aa1bf0c290417a8e1c1390cb1a WatchSource:0}: Error finding container 59805563b901f5adace385924880f9fa196174aa1bf0c290417a8e1c1390cb1a: Status 404 returned error can't find the container with id 59805563b901f5adace385924880f9fa196174aa1bf0c290417a8e1c1390cb1a
Oct 31 18:21:31.510446 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:31.509743    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:31.591355 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:31.591320    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq" event={"ID":"c8f7cf5d-858f-4ae2-b53e-c64881398c3f","Type":"ContainerStarted","Data":"59805563b901f5adace385924880f9fa196174aa1bf0c290417a8e1c1390cb1a"}
Oct 31 18:21:31.593182 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:31.593155    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerStarted","Data":"8b7fff8965badd296b83c5c42e37cd3cbc19e1263f5aeb185d9c54612146838a"}
Oct 31 18:21:31.595224 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:31.595199    2571 generic.go:356] "Generic (PLEG): container finished" podID="83fe51c6-ee5f-4ccf-a934-6d42a27d475b" containerID="5c52af85dabc9e2725ce1a088597e26758994da56c1eed5cdf4f53438555c300" exitCode=0
Oct 31 18:21:31.595905 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:31.595882    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgkm2" event={"ID":"83fe51c6-ee5f-4ccf-a934-6d42a27d475b","Type":"ContainerDied","Data":"5c52af85dabc9e2725ce1a088597e26758994da56c1eed5cdf4f53438555c300"}
Oct 31 18:21:31.607675 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:31.607647    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv" event={"ID":"445c9865-e1dc-467f-bb6f-f29e8c6b7f68","Type":"ContainerStarted","Data":"39033c4fc7c8c3ac2605e6fb163e03f9e0f38c2e146d1be0c16578288eb64fd1"}
Oct 31 18:21:31.745766 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:31.745714    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv" podStartSLOduration=2.434118069 podStartE2EDuration="4.745694484s" totalImagesPullingTime="2.311576415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:31.745107978 +0000 UTC m=+70.931774116" watchObservedRunningTime="2026-10-31 18:21:31.745694484 +0000 UTC m=+70.932360580"
Oct 31 18:21:32.623028 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.622776    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5" event={"ID":"65c82ec5-b112-4e22-9e89-25b3a69fe6a2","Type":"ContainerStarted","Data":"026ed57c28a734114a4014bf877f074f25d936009d2ac18bc3f5ef58db9ea1fe"}
Oct 31 18:21:32.623028 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.622840    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5" event={"ID":"65c82ec5-b112-4e22-9e89-25b3a69fe6a2","Type":"ContainerStarted","Data":"e1fc9bd6e79ff85797ccd0b2db3b9bf13262a07e1d28aa01a9646deda6b50c95"}
Oct 31 18:21:32.623028 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.622860    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5" event={"ID":"65c82ec5-b112-4e22-9e89-25b3a69fe6a2","Type":"ContainerStarted","Data":"dfb9e5281ada20fbc2f530d3e7814c53d1a776fe6b4da7120ef1462f78cefbb4"}
Oct 31 18:21:32.632360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.632334    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgkm2" event={"ID":"83fe51c6-ee5f-4ccf-a934-6d42a27d475b","Type":"ContainerStarted","Data":"b3d049f417235e1ab9931e10e39b71d46d691509bbeb59af67b0e568570e77c1"}
Oct 31 18:21:32.632508 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.632425    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgkm2" event={"ID":"83fe51c6-ee5f-4ccf-a934-6d42a27d475b","Type":"ContainerStarted","Data":"5f2e5683f42c14072215f7e7bb6275ef378699271abc68a68c8a61e854495ed4"}
Oct 31 18:21:32.752529 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.752041    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5" podStartSLOduration=2.950164161 podStartE2EDuration="5.752022392s" totalImagesPullingTime="2.801858231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:32.750509321 +0000 UTC m=+71.937175414" watchObservedRunningTime="2026-10-31 18:21:32.752022392 +0000 UTC m=+71.938688488"
Oct 31 18:21:32.762192 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.762162    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7ff57c786f-q298q"]
Oct 31 18:21:32.764497 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.764445    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.817321 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.817175    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 18:21:32.817321 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.817273    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 18:21:32.824525 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.824436    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ff57c786f-q298q"]
Oct 31 18:21:32.844050 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.844020    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-client-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.844213 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.844059    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.844213 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.844102    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-metrics-server-audit-profiles\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.844213 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.844132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cb3b9ece-b002-4973-95ed-862cd33ca046-audit-log\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.844213 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.844174    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-client-certs\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.844213 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.844204    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4cn\" (UniqueName: \"kubernetes.io/projected/cb3b9ece-b002-4973-95ed-862cd33ca046-kube-api-access-7p4cn\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.844505 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.844242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-tls\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.856639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.856250    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 18:21:32.856639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.856419    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1eblk8drb2j08\""
Oct 31 18:21:32.856639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.856520    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fk2vk\""
Oct 31 18:21:32.856639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.856576    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 18:21:32.867695 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.867056    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fgkm2" podStartSLOduration=3.748778429 podStartE2EDuration="5.867039524s" totalImagesPullingTime="2.02501894s" totalInitContainerRuntime="93.242155ms" isStatefulPod=true podCreationTimestamp="2026-10-31 18:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:32.864894603 +0000 UTC m=+72.051560708" watchObservedRunningTime="2026-10-31 18:21:32.867039524 +0000 UTC m=+72.053705617"
Oct 31 18:21:32.945497 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.945431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-metrics-server-audit-profiles\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.945497 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.945497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cb3b9ece-b002-4973-95ed-862cd33ca046-audit-log\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.945700 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.945541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-client-certs\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.945700 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.945572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4cn\" (UniqueName: \"kubernetes.io/projected/cb3b9ece-b002-4973-95ed-862cd33ca046-kube-api-access-7p4cn\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.945700 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.945599    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-tls\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.945700 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.945652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-client-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.945700 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.945678    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.946336 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.946314    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.947052 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.947027    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-metrics-server-audit-profiles\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.947663 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.947299    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cb3b9ece-b002-4973-95ed-862cd33ca046-audit-log\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.955045 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.955000    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-client-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.958019 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.957973    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-client-certs\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:32.963470 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:32.963427    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-tls\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:33.007955 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.007921    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4cn\" (UniqueName: \"kubernetes.io/projected/cb3b9ece-b002-4973-95ed-862cd33ca046-kube-api-access-7p4cn\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:33.080598 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.080561    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:33.541505 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.541348    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ff57c786f-q298q"]
Oct 31 18:21:33.548156 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:33.548120    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3b9ece_b002_4973_95ed_862cd33ca046.slice/crio-a131e89a883509aa1436e62d02354c7b2a4801e70a820d12a4987a78ebd2fd61 WatchSource:0}: Error finding container a131e89a883509aa1436e62d02354c7b2a4801e70a820d12a4987a78ebd2fd61: Status 404 returned error can't find the container with id a131e89a883509aa1436e62d02354c7b2a4801e70a820d12a4987a78ebd2fd61
Oct 31 18:21:33.625399 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.625348    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"]
Oct 31 18:21:33.631057 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.631039    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.635491 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.635411    2571 generic.go:356] "Generic (PLEG): container finished" podID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerID="23fb516d57f49fe77b6344c8f6d890a0ecdd2ac64f904ad7d452fedfcfb8c089" exitCode=0
Oct 31 18:21:33.636614 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.635632    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerDied","Data":"23fb516d57f49fe77b6344c8f6d890a0ecdd2ac64f904ad7d452fedfcfb8c089"}
Oct 31 18:21:33.639013 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.638615    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ff57c786f-q298q" event={"ID":"cb3b9ece-b002-4973-95ed-862cd33ca046","Type":"ContainerStarted","Data":"a131e89a883509aa1436e62d02354c7b2a4801e70a820d12a4987a78ebd2fd61"}
Oct 31 18:21:33.684974 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.684932    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 18:21:33.685309 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.685287    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 18:21:33.686258 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.685521    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 18:21:33.686258 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.685580    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lvpkw\""
Oct 31 18:21:33.686258 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.685657    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 18:21:33.686258 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.685908    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 18:21:33.686258 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.686228    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 18:21:33.693221 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.693155    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"]
Oct 31 18:21:33.755360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.753824    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.755360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.753879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-federate-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.755360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.753915    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.755360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.753967    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-serving-certs-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.755360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.754035    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fz2x\" (UniqueName: \"kubernetes.io/projected/884384ea-01db-45f2-8c04-2163de1950ca-kube-api-access-5fz2x\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.755360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.754087    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.755360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.754107    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-metrics-client-ca\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.755360 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.754163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.855395 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.855315    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.855395 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.855387    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.855639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.855431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-federate-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.855639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.855487    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.855639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.855537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-serving-certs-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.855639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.855567    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fz2x\" (UniqueName: \"kubernetes.io/projected/884384ea-01db-45f2-8c04-2163de1950ca-kube-api-access-5fz2x\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.855639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.855601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.855639 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.855626    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-metrics-client-ca\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.856393 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.856369    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-metrics-client-ca\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.857006 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.856979    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.858229 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.858087    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-serving-certs-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.862807 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.862728    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.863955 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.863911    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.864843 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.864797    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.865008 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.864985    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-federate-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.923070 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.923043    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fz2x\" (UniqueName: \"kubernetes.io/projected/884384ea-01db-45f2-8c04-2163de1950ca-kube-api-access-5fz2x\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:33.967150 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:33.966863    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:34.415697 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:34.415601    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"]
Oct 31 18:21:34.422608 ip-10-0-132-99 kubenswrapper[2571]: W1031 18:21:34.422566    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884384ea_01db_45f2_8c04_2163de1950ca.slice/crio-8fa123315d2dbc4848c9fc1f7961c421945e7dc165e4f3f8c1ccc2c2a1c8e467 WatchSource:0}: Error finding container 8fa123315d2dbc4848c9fc1f7961c421945e7dc165e4f3f8c1ccc2c2a1c8e467: Status 404 returned error can't find the container with id 8fa123315d2dbc4848c9fc1f7961c421945e7dc165e4f3f8c1ccc2c2a1c8e467
Oct 31 18:21:34.645901 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:34.645862    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4" event={"ID":"884384ea-01db-45f2-8c04-2163de1950ca","Type":"ContainerStarted","Data":"8fa123315d2dbc4848c9fc1f7961c421945e7dc165e4f3f8c1ccc2c2a1c8e467"}
Oct 31 18:21:34.648411 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:34.648381    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ztv9x" event={"ID":"3ad94d5c-d364-4d68-8795-70af86c61c4e","Type":"ContainerStarted","Data":"8abc3138cede6ef92cb491c82e37744e57da69e2d791746b1cb5a468c7613d78"}
Oct 31 18:21:34.649203 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:34.649180    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:34.765397 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:34.765286    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ztv9x" podStartSLOduration=67.314575897 podStartE2EDuration="1m13.765266648s" totalImagesPullingTime="6.450690751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:34.762022591 +0000 UTC m=+73.948688686" watchObservedRunningTime="2026-10-31 18:21:34.765266648 +0000 UTC m=+73.951932738"
Oct 31 18:21:35.665984 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:35.665951    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq" event={"ID":"c8f7cf5d-858f-4ae2-b53e-c64881398c3f","Type":"ContainerStarted","Data":"da5befbf5552c2bd6022392d854413a2ee69b4cb48218987c2da1eb4f37c2c90"}
Oct 31 18:21:36.682550 ip-10-0-132-99 kubenswrapper[2571]: I1031 18:21:36.681529    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m56gf" event={"ID":"28594e99-2b75-44a6-8922-315e6b680700","Type":"ContainerStarted","Data":"64b9d57c6615623b96183bc52593ce4b1573b3e3ebecc4e352baa732368060bb"}
Oct 31 18:21:36.683893 ip-10-0-132-99 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 18:21:36.705930 ip-10-0-132-99 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 18:21:36.706164 ip-10-0-132-99 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 18:21:36.706394 ip-10-0-132-99 systemd[1]: kubelet.service: Consumed 4.150s CPU time.
Oct 31 18:21:36.714129 ip-10-0-132-99 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 18:21:36.806832 ip-10-0-132-99 kubenswrapper[5551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:21:36.806832 ip-10-0-132-99 kubenswrapper[5551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 18:21:36.806832 ip-10-0-132-99 kubenswrapper[5551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:21:36.806832 ip-10-0-132-99 kubenswrapper[5551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:21:36.809055 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.808977    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:21:36.809055 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809054    5551 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:21:36.809055 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809057    5551 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809060    5551 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809064    5551 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809066    5551 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809069    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809072    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809075    5551 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809079    5551 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809081    5551 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809084    5551 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809086    5551 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809089    5551 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809092    5551 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809095    5551 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809098    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809100    5551 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809103    5551 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809105    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809108    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:21:36.809164 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809110    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809113    5551 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809116    5551 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809126    5551 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809129    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809131    5551 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809135    5551 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809139    5551 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809142    5551 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809145    5551 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809147    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809150    5551 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809152    5551 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809155    5551 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809159    5551 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809162    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809165    5551 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809167    5551 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809170    5551 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:21:36.809803 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809173    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809176    5551 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809178    5551 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809181    5551 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809184    5551 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809186    5551 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809188    5551 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809191    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809194    5551 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809196    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809199    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809201    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809204    5551 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809206    5551 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809208    5551 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809212    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809216    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809219    5551 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809221    5551 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809224    5551 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:21:36.810308 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809226    5551 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809229    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809231    5551 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809234    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809236    5551 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809239    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809242    5551 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809244    5551 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809247    5551 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809249    5551 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809251    5551 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809254    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809256    5551 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809259    5551 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809261    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809263    5551 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809266    5551 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809268    5551 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809270    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809273    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:21:36.810794 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809275    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809278    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809280    5551 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809283    5551 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809286    5551 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809288    5551 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809291    5551 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809294    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809297    5551 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809301    5551 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809304    5551 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809307    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809310    5551 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809313    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809315    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809319    5551 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809321    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809324    5551 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809327    5551 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809330    5551 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:21:36.811286 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809332    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809335    5551 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809337    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809348    5551 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809352    5551 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.809355    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810191    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810200    5551 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810204    5551 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810207    5551 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810210    5551 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810213    5551 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810216    5551 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810219    5551 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810222    5551 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810225    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810227    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810230    5551 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810233    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810235    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:21:36.811776 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810237    5551 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810240    5551 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810243    5551 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810245    5551 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810248    5551 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810250    5551 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810253    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810255    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810258    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810261    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810263    5551 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810266    5551 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810268    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810271    5551 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810274    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810276    5551 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810279    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810282    5551 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810284    5551 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:21:36.812254 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810286    5551 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810289    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810291    5551 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810294    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810296    5551 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810299    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810301    5551 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810304    5551 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810306    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810309    5551 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810311    5551 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810314    5551 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810317    5551 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810319    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810322    5551 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810331    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810334    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810337    5551 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810339    5551 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:21:36.812765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810342    5551 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810344    5551 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810347    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810349    5551 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810352    5551 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810355    5551 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810357    5551 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810374    5551 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810378    5551 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810381    5551 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810384    5551 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810387    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810389    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810394    5551 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810398    5551 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810401    5551 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810404    5551 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810406    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810409    5551 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810411    5551 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:21:36.813217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810414    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810416    5551 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810418    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810421    5551 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810423    5551 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810426    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810429    5551 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810431    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810434    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810437    5551 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810439    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810442    5551 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810444    5551 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810447    5551 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810449    5551 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810471    5551 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810476    5551 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810481    5551 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810484    5551 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810486    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:21:36.813733 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810489    5551 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810492    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810494    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810497    5551 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810500    5551 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810502    5551 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810504    5551 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810507    5551 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810510    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810512    5551 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810514    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810518    5551 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810522    5551 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.810525    5551 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811034    5551 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811040    5551 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811043    5551 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811046    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811050    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:21:36.814198 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811053    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811056    5551 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811060    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811062    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811065    5551 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811068    5551 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811071    5551 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811074    5551 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811076    5551 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811078    5551 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811081    5551 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811084    5551 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811086    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811089    5551 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811091    5551 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811094    5551 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811096    5551 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811099    5551 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811101    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811104    5551 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:21:36.814675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811106    5551 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811109    5551 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811111    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811114    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811116    5551 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811119    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811122    5551 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811124    5551 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811127    5551 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811129    5551 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811132    5551 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811135    5551 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811137    5551 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811140    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811142    5551 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811146    5551 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811148    5551 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811151    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811153    5551 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811156    5551 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:21:36.815167 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811158    5551 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811161    5551 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811163    5551 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811166    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811168    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811171    5551 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811173    5551 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811176    5551 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811178    5551 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811181    5551 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811183    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811186    5551 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811188    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811191    5551 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811193    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811197    5551 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811201    5551 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811204    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811207    5551 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:21:36.815656 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811209    5551 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811212    5551 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811214    5551 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811217    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811220    5551 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811223    5551 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811225    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811228    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811231    5551 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811235    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811239    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811242    5551 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811244    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811247    5551 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811249    5551 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811252    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811254    5551 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811257    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811259    5551 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:21:36.816092 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811262    5551 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811267    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811269    5551 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811272    5551 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811274    5551 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811277    5551 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811279    5551 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811282    5551 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811284    5551 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811287    5551 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811289    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811292    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811295    5551 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811298    5551 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811300    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811304    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811306    5551 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811309    5551 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811311    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:21:36.816595 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811314    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811316    5551 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811319    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.811321    5551 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811416    5551 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811426    5551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811435    5551 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811440    5551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811444    5551 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811447    5551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811451    5551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811473    5551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811476    5551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811479    5551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811485    5551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811488    5551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811491    5551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811493    5551 flags.go:64] FLAG: --cgroup-root=""
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811496    5551 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811499    5551 flags.go:64] FLAG: --client-ca-file=""
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811504    5551 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811507    5551 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811512    5551 flags.go:64] FLAG: --cluster-domain=""
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811514    5551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 18:21:36.817030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811518    5551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811521    5551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811524    5551 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811527    5551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811530    5551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811534    5551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811537    5551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811539    5551 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811542    5551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811545    5551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811549    5551 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811552    5551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811556    5551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811559    5551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811562    5551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811565    5551 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811568    5551 flags.go:64] FLAG: --enable-server="true"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811570    5551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811574    5551 flags.go:64] FLAG: --event-burst="100"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811577    5551 flags.go:64] FLAG: --event-qps="50"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811580    5551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811583    5551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811587    5551 flags.go:64] FLAG: --eviction-hard=""
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811591    5551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 18:21:36.817971 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811594    5551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811597    5551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811600    5551 flags.go:64] FLAG: --eviction-soft=""
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811603    5551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811606    5551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811609    5551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811612    5551 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811615    5551 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811618    5551 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811621    5551 flags.go:64] FLAG: --feature-gates=""
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811625    5551 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811628    5551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811631    5551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811634    5551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811637    5551 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811640    5551 flags.go:64] FLAG: --help="false"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811642    5551 flags.go:64] FLAG: --hostname-override="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811646    5551 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811649    5551 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811652    5551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811655    5551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811658    5551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811661    5551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 18:21:36.818793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811665    5551 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811667    5551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811670    5551 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811673    5551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811676    5551 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811679    5551 flags.go:64] FLAG: --kube-reserved=""
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811682    5551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811686    5551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811689    5551 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811692    5551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811694    5551 flags.go:64] FLAG: --lock-file=""
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811697    5551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811700    5551 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811704    5551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811709    5551 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811712    5551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811715    5551 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811718    5551 flags.go:64] FLAG: --logging-format="text"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811720    5551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811724    5551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811727    5551 flags.go:64] FLAG: --manifest-url=""
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811730    5551 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811734    5551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811737    5551 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811740    5551 flags.go:64] FLAG: --max-pods="110"
Oct 31 18:21:36.819336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811743    5551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811746    5551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811766    5551 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811769    5551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811773    5551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811776    5551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811779    5551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811786    5551 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811790    5551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811793    5551 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811796    5551 flags.go:64] FLAG: --pod-cidr=""
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811799    5551 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811801    5551 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811804    5551 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811809    5551 flags.go:64] FLAG: --port="10250"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811811    5551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811814    5551 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b6db748a16ba5b95"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811817    5551 flags.go:64] FLAG: --qos-reserved=""
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811820    5551 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811823    5551 flags.go:64] FLAG: --register-node="true"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811827    5551 flags.go:64] FLAG: --register-with-taints=""
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811831    5551 flags.go:64] FLAG: --registry-burst="10"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811834    5551 flags.go:64] FLAG: --registry-qps="5"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811837    5551 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811840    5551 flags.go:64] FLAG: --reserved-memory=""
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811844    5551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 18:21:36.820001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811847    5551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811850    5551 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811853    5551 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811856    5551 flags.go:64] FLAG: --runonce="false"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811859    5551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811862    5551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811865    5551 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811867    5551 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811870    5551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811873    5551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811876    5551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811879    5551 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811882    5551 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811884    5551 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811887    5551 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811890    5551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811893    5551 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811896    5551 flags.go:64] FLAG: --system-cgroups=""
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811898    5551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811903    5551 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811907    5551 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811910    5551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811914    5551 flags.go:64] FLAG: --tls-min-version=""
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811917    5551 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811920    5551 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 18:21:36.820663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811922    5551 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811927    5551 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811930    5551 flags.go:64] FLAG: --v="2"
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811934    5551 flags.go:64] FLAG: --version="false"
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811938    5551 flags.go:64] FLAG: --vmodule=""
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811942    5551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.811945    5551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812074    5551 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812078    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812081    5551 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812084    5551 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812087    5551 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812089    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812092    5551 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812095    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812097    5551 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812100    5551 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812102    5551 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812105    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812107    5551 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812110    5551 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:21:36.821257 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812112    5551 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812116    5551 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812119    5551 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812121    5551 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812124    5551 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812126    5551 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812130    5551 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812133    5551 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812135    5551 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812138    5551 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812140    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812143    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812146    5551 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812149    5551 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812151    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812154    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812156    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812159    5551 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812161    5551 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812164    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:21:36.822019 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812166    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812169    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812171    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812174    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812176    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812179    5551 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812181    5551 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812184    5551 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812186    5551 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812189    5551 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812191    5551 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812194    5551 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812196    5551 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812200    5551 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812205    5551 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812208    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812211    5551 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812214    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812218    5551 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:21:36.822583 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812220    5551 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812223    5551 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812225    5551 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812228    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812230    5551 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812234    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812236    5551 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812239    5551 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812241    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812244    5551 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812246    5551 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812248    5551 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812251    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812253    5551 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812256    5551 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812259    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812261    5551 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812264    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812266    5551 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812269    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:21:36.823205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812272    5551 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812274    5551 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812277    5551 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812279    5551 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812282    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812284    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812287    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812289    5551 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812295    5551 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812298    5551 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812300    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812304    5551 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812306    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812310    5551 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812313    5551 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812316    5551 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812319    5551 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812324    5551 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812326    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:21:36.823786 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812329    5551 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812332    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812335    5551 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812337    5551 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812340    5551 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812342    5551 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812344    5551 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812347    5551 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812349    5551 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812352    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812354    5551 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812356    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812359    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812361    5551 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.812387    5551 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812421    5551 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:21:36.824246 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812425    5551 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812428    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812431    5551 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812434    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812436    5551 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812438    5551 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812441    5551 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812445    5551 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812448    5551 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812450    5551 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812469    5551 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812473    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812475    5551 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812478    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812482    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812484    5551 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812487    5551 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812489    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812492    5551 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812494    5551 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:21:36.824744 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812497    5551 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812499    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812502    5551 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812504    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812507    5551 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812509    5551 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812512    5551 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812514    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812516    5551 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812519    5551 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812521    5551 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812524    5551 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812526    5551 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812529    5551 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812531    5551 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812533    5551 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812535    5551 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812538    5551 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812540    5551 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:21:36.825234 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812543    5551 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812546    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812548    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812550    5551 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812553    5551 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812555    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812558    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812565    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812567    5551 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812570    5551 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812572    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812575    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812577    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812580    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812582    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812584    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812587    5551 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812589    5551 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812592    5551 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812594    5551 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:21:36.825740 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812597    5551 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812599    5551 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812602    5551 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812604    5551 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812608    5551 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812612    5551 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812615    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812618    5551 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812620    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812622    5551 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812625    5551 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812627    5551 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812630    5551 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812633    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812635    5551 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812637    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812642    5551 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812644    5551 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812646    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:21:36.826212 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812650    5551 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812652    5551 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812655    5551 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812657    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812660    5551 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812663    5551 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812666    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812668    5551 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812671    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812673    5551 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812676    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812679    5551 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812681    5551 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812684    5551 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812686    5551 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812689    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812691    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812693    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812696    5551 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:21:36.826765 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812698    5551 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812701    5551 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812703    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812705    5551 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812708    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812712    5551 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812715    5551 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.812718    5551 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.816361    5551 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.816376    5551 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.816391    5551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.816395    5551 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816448    5551 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816469    5551 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816473    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816476    5551 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816479    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816482    5551 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816485    5551 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:21:36.827205 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816488    5551 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816490    5551 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816493    5551 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816495    5551 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816498    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816500    5551 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816504    5551 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816506    5551 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816509    5551 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816511    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816514    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816516    5551 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816519    5551 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816521    5551 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816524    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816527    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816529    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816531    5551 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816534    5551 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:21:36.827668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816536    5551 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816539    5551 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816541    5551 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816544    5551 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816547    5551 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816552    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816554    5551 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816557    5551 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816560    5551 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816563    5551 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816565    5551 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816568    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816570    5551 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816573    5551 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816575    5551 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816577    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816580    5551 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816582    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816585    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816587    5551 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:21:36.828145 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816590    5551 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816593    5551 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816596    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816599    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816602    5551 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816604    5551 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816607    5551 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816610    5551 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816612    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816614    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816617    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816619    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816622    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816625    5551 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816627    5551 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816630    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816632    5551 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816635    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816637    5551 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:21:36.828664 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816639    5551 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816642    5551 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816644    5551 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816647    5551 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816650    5551 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816652    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816654    5551 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816657    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816659    5551 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816661    5551 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816664    5551 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816666    5551 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816669    5551 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816671    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816674    5551 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816677    5551 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816679    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816682    5551 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816684    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816687    5551 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:21:36.829156 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816689    5551 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816692    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816694    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816696    5551 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816699    5551 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816701    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816704    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816706    5551 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816708    5551 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816712    5551 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816716    5551 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816719    5551 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816721    5551 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816724    5551 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816726    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816729    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816731    5551 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816733    5551 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816736    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:21:36.829708 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816738    5551 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816740    5551 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.816767    5551 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816900    5551 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816903    5551 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816906    5551 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816908    5551 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816911    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816914    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816916    5551 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816919    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816921    5551 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816924    5551 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816927    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816929    5551 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816931    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:21:36.830206 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816934    5551 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816936    5551 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816938    5551 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816941    5551 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816943    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816946    5551 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816949    5551 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816952    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816955    5551 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816958    5551 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816961    5551 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816963    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816968    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816970    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816973    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816975    5551 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816978    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816980    5551 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816983    5551 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:21:36.830718 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816985    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816988    5551 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816990    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816993    5551 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816995    5551 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.816997    5551 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817034    5551 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817082    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817443    5551 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817451    5551 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817472    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817477    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817482    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817486    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817491    5551 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817498    5551 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817504    5551 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817509    5551 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817514    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:21:36.831228 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817518    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817528    5551 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817532    5551 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817537    5551 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817542    5551 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817546    5551 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817553    5551 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817558    5551 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817562    5551 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817566    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817570    5551 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817575    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817579    5551 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817589    5551 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817592    5551 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817597    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817601    5551 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817606    5551 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817610    5551 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817614    5551 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:21:36.831789 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817620    5551 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817625    5551 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817629    5551 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817633    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817638    5551 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817642    5551 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817651    5551 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817655    5551 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817659    5551 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817663    5551 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817667    5551 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817671    5551 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817675    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817680    5551 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817684    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817688    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817692    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817697    5551 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817705    5551 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817714    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:21:36.832484 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817718    5551 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817723    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817727    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817731    5551 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817736    5551 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817740    5551 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817745    5551 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817749    5551 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817754    5551 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817758    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817762    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817771    5551 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817775    5551 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817781    5551 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817785    5551 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:21:36.832980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.817849    5551 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817958    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817972    5551 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817977    5551 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817982    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817987    5551 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817991    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817995    5551 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.817999    5551 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818003    5551 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818007    5551 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818011    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818016    5551 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818022    5551 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818034    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818039    5551 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818044    5551 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818048    5551 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818052    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:21:36.833336 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818057    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818061    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818065    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818069    5551 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818073    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818077    5551 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818081    5551 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818101    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818106    5551 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818110    5551 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818115    5551 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818121    5551 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818125    5551 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818129    5551 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818134    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818138    5551 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818142    5551 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818146    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818151    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:21:36.833838 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818155    5551 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818177    5551 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818299    5551 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818308    5551 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818312    5551 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818315    5551 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818317    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818320    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818322    5551 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818325    5551 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818330    5551 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818333    5551 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818335    5551 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818337    5551 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818340    5551 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818342    5551 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818345    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818347    5551 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818350    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818352    5551 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:21:36.834283 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818354    5551 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818357    5551 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818361    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818363    5551 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818366    5551 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818368    5551 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818375    5551 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818377    5551 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818380    5551 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818382    5551 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818385    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818387    5551 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818389    5551 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818392    5551 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818394    5551 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818397    5551 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818399    5551 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818401    5551 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818404    5551 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818406    5551 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:21:36.834771 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818409    5551 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818413    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818417    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818421    5551 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818424    5551 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818427    5551 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818429    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818432    5551 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818434    5551 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818437    5551 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818442    5551 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818446    5551 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818448    5551 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818451    5551 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818478    5551 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818481    5551 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818484    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818486    5551 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818491    5551 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:21:36.835261 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818494    5551 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818497    5551 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818499    5551 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818502    5551 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818504    5551 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818507    5551 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818509    5551 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818512    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818514    5551 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:36.818517    5551 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.818586    5551 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.819539    5551 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.819593    5551 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.819769    5551 server.go:1014] "Starting client certificate rotation"
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.819917    5551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.819977    5551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 18:15:21 +0000 UTC" deadline="2028-07-28 18:15:17.867253746 +0000 UTC"
Oct 31 18:21:36.835761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.820012    5551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15263h53m41.047244501s"
Oct 31 18:21:36.836150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.820261    5551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 18:21:36.836150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.821280    5551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 18:21:36.836150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.822486    5551 log.go:25] "Validated CRI v1 runtime API"
Oct 31 18:21:36.836150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.824328    5551 log.go:25] "Validated CRI v1 image API"
Oct 31 18:21:36.836150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.825046    5551 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 18:21:36.836150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.827113    5551 fs.go:135] Filesystem UUIDs: map[6044eb23-9831-4de4-8c85-af58aa0d139f:/dev/nvme0n1p4 710be49e-b8cb-4e11-916d-abe5e7617268:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 31 18:21:36.836357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.827128    5551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0097d97e1ef0c7adeaeb69eb26f8808d654557ad46b60bbb3bdf47bf65d84532/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0097d97e1ef0c7adeaeb69eb26f8808d654557ad46b60bbb3bdf47bf65d84532/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/018a1eaae75ee0d8c6ef2cd2fabc62a6e253fc6f8a0ac41eeafe39536af77525/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/018a1eaae75ee0d8c6ef2cd2fabc62a6e253fc6f8a0ac41eeafe39536af77525/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/093b014626279e61080b83681b46d60c99de3ef65c3f45de8aad99340e5d5c23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/093b014626279e61080b83681b46d60c99de3ef65c3f45de8aad99340e5d5c23/userdata/shm major:0 minor:280 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e99dc45621c676dae190438818cba7c89449d024ff30eb3ba5e322562f8399d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e99dc45621c676dae190438818cba7c89449d024ff30eb3ba5e322562f8399d/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2105f6e721f592c0f6014006b097116a9857285c6406f46010196dd957d790f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2105f6e721f592c0f6014006b097116a9857285c6406f46010196dd957d790f1/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3dac68756a73a04f709f12b3d6c1eefa758c545abf17718f7b5a361f7e6cf4f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3dac68756a73a04f709f12b3d6c1eefa758c545abf17718f7b5a361f7e6cf4f4/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f5039aa0259af9d5c470852b5c56ce454f9100a23979e263cb35a0ba4ec0b8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f5039aa0259af9d5c470852b5c56ce454f9100a23979e263cb35a0ba4ec0b8d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b7695f34b7a55d47f3bb55a9990a93090dfd794267f1557d28f2ff1aa26a225/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b7695f34b7a55d47f3bb55a9990a93090dfd794267f1557d28f2ff1aa26a225/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59110ac180d05d3383f18e51c3d2ef56424f0c974f43cd1b6e21bbcd044c6e28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59110ac180d05d3383f18e51c3d2ef56424f0c974f43cd1b6e21bbcd044c6e28/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59805563b901f5adace385924880f9fa196174aa1bf0c290417a8e1c1390cb1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59805563b901f5adace385924880f9fa196174aa1bf0c290417a8e1c1390cb1a/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82cbbb05cb44ecbf8c2dc8e3117ed3bd8fca4cc4c15893a923eddacb4159dbb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82cbbb05cb44ecbf8c2dc8e3117ed3bd8fca4cc4c15893a923eddacb4159dbb3/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89b6ce0535b96eba824e61d9d6c38ee6d1969cbb71aed8545acdd294c558236d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89b6ce0535b96eba824e61d9d6c38ee6d1969cbb71aed8545acdd294c558236d/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b7fff8965badd296b83c5c42e37cd3cbc19e1263f5aeb185d9c54612146838a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b7fff8965badd296b83c5c42e37cd3cbc19e1263f5aeb185d9c54612146838a/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8df7b013ba992048504b55b5954e8649371266a700cbc03b849d4b0604a8d704/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8df7b013ba992048504b55b5954e8649371266a700cbc03b849d4b0604a8d704/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fa123315d2dbc4848c9fc1f7961c421945e7dc165e4f3f8c1ccc2c2a1c8e467/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fa123315d2dbc4848c9fc1f7961c421945e7dc165e4f3f8c1ccc2c2a1c8e467/userdata/shm major:0 minor:413 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a00ba90066290cc95585207e999b86e0b3da4b2b3aa11f4afd5dce756983b2f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a00ba90066290cc95585207e999b86e0b3da4b2b3aa11f4afd5dce756983b2f7/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a131e89a883509aa1436e62d02354c7b2a4801e70a820d12a4987a78ebd2fd61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a131e89a883509aa1436e62d02354c7b2a4801e70a820d12a4987a78ebd2fd61/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aacf2191f4fd9790f551b62643322993358b8dd5805ed54198a9dda1ec8bc97d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aacf2191f4fd9790f551b62643322993358b8dd5805ed54198a9dda1ec8bc97d/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac28dc82845f0baae359492484e9bb817e94a43c75317221fe99aa941d1c0a11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac28dc82845f0baae359492484e9bb817e94a43c75317221fe99aa941d1c0a11/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3a917921a0754afb3d482b95d56520d071e371af584de5cc2278953ca1e216a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3a917921a0754afb3d482b95d56520d071e371af584de5cc2278953ca1e216a/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce0d6948e36ad007965b74f660246b4f836c2b0cd11ee3c75e7136b268d39bca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce0d6948e36ad007965b74f660246b4f836c2b0cd11ee3c75e7136b268d39bca/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cee41971d4a8889f0ee9c6c672b2e0710c00d4c16c44b98d31ff9d3deeac4cb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cee41971d4a8889f0ee9c6c672b2e0710c00d4c16c44b98d31ff9d3deeac4cb9/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d24d184aa2ec9bcac439f2d949c9945598897bc7a9cfdaa271a427f0f703e74c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d24d184aa2ec9bcac439f2d949c9945598897bc7a9cfdaa271a427f0f703e74c/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea3ab49407dac02aad13cc91281751709100163ad3a967086ca9df9775e0b579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea3ab49407dac02aad13cc91281751709100163ad3a967086ca9df9775e0b579/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb570da9627300e528f6a81428e9d8b7a91eb8e7a2557f2b41f573c3ac8168d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb570da9627300e528f6a81428e9d8b7a91eb8e7a2557f2b41f573c3ac8168d6/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd649c93eab2f98b3997f01c56d2ff0f5c5385a2f6e0066591c4387d7dff8d5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd649c93eab2f98b3997f01c56d2ff0f5c5385a2f6e0066591c4387d7dff8d5b/userdata/shm major:0 minor:275 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volumes/kubernetes.io~projected/kube-api-access-cprjg:{mountpoint:/var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volumes/kubernetes.io~projected/kube-api-access-cprjg major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12b631c1-447d-43ee-9f5a-d77bfb04e03c/volumes/kubernetes.io~projected/kube-api-access-mpzgw:{mountpoint:/var/lib/kubelet/pods/12b631c1-447d-43ee-9f5a-d77bfb04e03c/volumes/kubernetes.io~projected/kube-api-access-mpzgw major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fe78dfe-d1f0-4196-9832-702532afcaf8/volumes/kubernetes.io~projected/kube-api-access-czvds:{mountpoint:/var/lib/kubelet/pods/1fe78dfe-d1f0-4196-9832-702532afcaf8/volumes/kubernetes.io~projected/kube-api-access-czvds major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28594e99-2b75-44a6-8922-315e6b680700/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/28594e99-2b75-44a6-8922-315e6b680700/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~projected/kube-api-access-bk88q:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~projected/kube-api-access-bk88q major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~projected/tls-assets major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/config-volume major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/web-config major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2efd8852-abb6-40d2-afda-40fe91bbd18f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2efd8852-abb6-40d2-afda-40fe91bbd18f/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3386381a-8b41-473b-9400-04d3d3eb2c7e/volumes/kubernetes.io~projected/kube-api-access-mtb7n:{mountpoint:/var/lib/kubelet/pods/3386381a-8b41-473b-9400-04d3d3eb2c7e/volumes/kubernetes.io~projected/kube-api-access-mtb7n major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3497b74a-87c3-4f27-91b7-dac04680a875/volumes/kubernetes.io~projected/kube-api-access-6m5tp:{mountpoint:/var/lib/kubelet/pods/3497b74a-87c3-4f27-91b7-dac04680a875/volumes/kubernetes.io~projected/kube-api-access-6m5tp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ad94d5c-d364-4d68-8795-70af86c61c4e/volumes/kubernetes.io~projected/kube-api-access-8bbbk:{mountpoint:/var/lib/kubelet/pods/3ad94d5c-d364-4d68-8795-70af86c61c4e/volumes/kubernetes.io~projected/kube-api-access-8bbbk major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~projected/kube-api-access-sdx5n:{mountpoint:/var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~projected/kube-api-access-sdx5n major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~projected/kube-api-access-v62m6:{mountpoint:/var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~projected/kube-api-access-v62m6 major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~projected/kube-api-access-bp7jq:{mountpoint:/var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~projected/kube-api-access-bp7jq major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~projected/kube-api-access-v6bg2:{mountpoint:/var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~projected/kube-api-access-v6bg2 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69a446fa-5d09-4f26-9b5d-72f3fbe42064/volumes/kubernetes.io~projected/kube-api-access-8dchr:{mountpoint:/var/lib/kubelet/pods/69a446fa-5d09-4f26-9b5d-72f3fbe42064/volumes/kubernetes.io~projected/kube-api-access-8dchr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69a446fa-5d09-4f26-9b5d-72f3fbe42064/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/69a446fa-5d09-4f26-9b5d-72f3fbe42064/volumes/kubernetes.io~secret/metrics-tls major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~projected/kube-api-access-jzp8k:{mountpoint:/var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~projected/kube-api-access-jzp8k major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~projected/kube-api-access-5fz2x:{mountpoint:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~projected/kube-api-access-5fz2x major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b37353a-7cbf-478e-b7c5-83a71f6c1986/volumes/kubernetes.io~projected/kube-api-access-qrt52:{mountpoint:/var/lib/kubelet/pods/8b37353a-7cbf-478e-b7c5-83a71f6c1986/volumes/kubernetes.io~projected/kube-api-access-qrt52 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~projected/kube-api-access-mmw9b:{mountpoint:/var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~projected/kube-api-access-mmw9b major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97e5d8ad-0765-4815-a495-53de9ca7b24b/volumes/kubernetes.io~projected/kube-api-access-zrltf:{mountpoint:/var/lib/kubelet/pods/97e5d8ad-0765-4815-a495-53de9ca7b24b/volumes/kubernetes.io~projected/kube-api-access-zrltf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97e5d8ad-0765-4815-a495-53de9ca7b24b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/97e5d8ad-0765-4815-a495-53de9ca7b24b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e2f392-7651-4d4d-9c09-b4978aabcebf/volumes/kubernetes.io~projected/kube-api-access-ldl6f:{mountpoint:/var/lib/kubelet/pods/a8e2f392-7651-4d4d-9c09-b4978aabcebf/volumes/kubernetes.io~projected/kube-api-access-ldl6f major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~projected/kube-api-access-whmgd:{mountpoint:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~projected/kube-api-access-whmgd major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~projected/kube-api-access-7p4cn:{mountpoint:/var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~projected/kube-api-access-7p4cn major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deca97c4-946e-4974-b30e-dd0595a94633/volumes/kubernetes.io~projected/kube-api-access-cpss5:{mountpoint:/var/lib/kubelet/pods/deca97c4-946e-4974-b30e-dd0595a94633/volumes/kubernetes.io~projected/kube-api-access-cpss5 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec058e17-0198-4c81-8ce4-2e3a616f9d69/volumes/kubernetes.io~projected/kube-api-access-jkcqf:{mountpoint:/var/lib/kubelet/pods/ec058e17-0198-4c81-8ce4-2e3a616f9d69/volumes/kubernetes.io~projected/kube-api-access-jkcqf major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec058e17-0198-4c81-8ce4-2e3a616f9d69/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ec058e17-0198-4c81-8ce4-2e3a616f9d69/volumes/kubernetes.io~secret/metrics-certs major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882/volumes/kubernetes.io~projected/kube-api-access-64wql:{mountpoint:/var/lib/kubelet/pods/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882/volumes/kubernetes.io~projected/kube-api-access-64wql major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cc08fc3ac08d9a01a663bbf6e1236047c28be52122f01422d5d11fd8c60b9f31/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4d1336cc34121c662908531d125d38e6ac4d77a0953b7ea961cedf49aeba2725/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/51aea5991bdee0e0841ee621472d6e646cde81b2746e5ca405edc754a2ae93f3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3ece76f5e21a4bb5de8a15f0ce7c7e47686e95e834cec3ff88c5892a8ffbde59/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/18cbc5907d7a2513f0c7a939d0db5b6d3f2ee062ce3d32d73974cd41fa1147ff/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/27a5e37809fa54e9b22a090aaea8dad41f10490487d4e82d55362a823481d59a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5131609c6ef10442718a4388409852da4317a3f82b418f18dbfe42fd7a47d699/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8e18fd990059db8e9a0858a4091fae3f137330f8ebebab067ccbc2fae22cbc45/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/4f61dd570ea63c5bc48ddafd9beabc1cd1c8bd51aa364ec587cb7be439b81c6a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/1f336d9c64cf838f25fba09cd8e959c8fc8cae08ea1307bc5bf971cdd5cee32c/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/942d23af2bcc03baba91d606b49ddafde883953f39969673293a7b5711b66a64/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/8d7554769b69552b90f159450dd78530ca7b2037eea01c02be910f3938fe5b18/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/112175e741edf96b8c4f02f16ebc68c41d025d758cda4462c2a75b96ada714cc/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/465630a748c9421753b2aeb69b5322fe373c5279ad9c6483500da3e87f49b88b/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/c71e6304db93ddafc1f597b40a4c3384c6e931e4a9f5bc9c716909840585b41c/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/dcfbb463a3e93323bc68663c845fe5b21d10aaa0bed07aded306969ec401c3b4/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/060c6cb895a93815d018a9920bccad3c081d2b21e96a46cc85f8e38421368357/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/a190d3196f1a54430fa0de72a79558c9a82b2bc1731dcb09eb71c414a370c2cc/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/546c3ceeb933e4ba47676f0982b137cbf3cef38040a47cf80249057077746d2e/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/be018db3b96d76762ba95d783a29df4c7972f6e0f813d8c9b1550d99d2b16dd6/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/3832916915a9effd0d5d5ba02f8a92567ad42203fa0261c58e5b43709c6fedd7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/80b9f25e1de523e35f36eec0d0b24a0121766e8f0c59b0617ada6301f541204f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/bed2787f01be64c5d5edd08aa61146f76380727d82a4e8422bdadf7ce41dde77/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d3563164a61e07c5159bd3a854c54141f35f6971a99fdf761c27feea2599b3eb/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/8bc841a15a04b69c7fe60e6d35a1fa6294535e21d012f3b0f0616b331a1b55f1/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/3ee75301c33144782ac4eec735dc4ce2b56e46d78f6cca7eee3e83d1eeac44b0/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d2fbbdb5ba573506d9f8b6d4f1abaa09dfdebd83cd94eb06c98ffb7a46872048/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/c73875e58c512440e4d084f5553400a827580b6e097e892935d9abecbe3424fb/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/744ee6a67e1d24c6c9542483dfd8f88e95ab70aa7b66ace07cf3451736dfa42d/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/ecf3c377b6102f16579aecff30d06bf27420ea2a0473b46a75d3973aada5b527/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/469b4ebd7bf4dcc27c144a0d6388304d015109f2ea317f25358c2bb4ba6efdec/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/33b0c3a1d2a2a8ee189f7669e1e7e9f1fee4194ae2a3603709b119323fe3aa3e/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/555042e0b744aa245498feee27d1e2d57907df019747d5cffae78159a7ae885e/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/5381b43804a7558af7a29f45d57e159c49bd0847441be1b923a63282dd676847/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/4649164dd4801417ba3feff72d7c297c03657d866f0a04f53665746e861a9a55/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/d2645d9a6d8590c003cf5397792789849dbbc4f4312aa0d03ae883926afd94e4/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/f36e77e54ef4ea9d28fd1e320c17260ae557218b236fd701c1f1996e2be30101/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/988360093d49d7d6cc7253feec69ebe841f9e55d72df1cba33aa8084939476b9/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/dc082256b534c7fb3d24da220972e8bf92f2f546590d92b72fc2d6a310f0b33f/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/3722c44c5502c4ac7361d3dcee28d32f7fc5a9073cab5eaa78052033478ca81a/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/c20dd5931c87d6f87631410d278b88ad8bed96a79f6f83b6b264f425593a62ee/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/be04eac3ab41b409793f85b1fc2956e81ee56abdb1763b7080e319ebe5214155/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/e78a16d5a3c7610a108544f3c66e4d6455ceef69a27aa36438e31f636abb1f01/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/14f4d2c95470731128139af15f6ac703cbc447a65b2966d5841adf3a57236bb5/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/91fee47e033d0a3c6b67aecc0fcd64d070cb42eecb717b65f3c6df733e09b694/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/069d42ea1ff2de3bf2c206a6aabbb1aaab836c01fb4cedfa27dc670af72cf17d/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/190a276ff92732af51f8261710bb646f629d580e3b61380bd3139012860a6930/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/4d319469aab6d239212737b7cb69aed10162a84e3864a12018cd3536a4ed7b55/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/1ef1ccf67397ebb3f0bba2c96286185cf7879f256c4c03be1117f944266ebf3d/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/c50db701c48eb9acebfa2ee47893546dddbe04426494313bbdee861fbf28786a/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/d122deae2230b549fe872baec856f337031dfc860b975cdfc89c1513458ba81f/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/bbe6af971d3c1cde283c0f5119ef20fa02e91a1853f549aab17532f9a99ecefb/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/da1da107518515c5ac0369f7236ea1e8a83a189b0f61747fc77051f04e730673/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/4a4e2c0b40a147a048a44bda8ec6e38d56d2512171cb18b5fa1c035341927d96/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/d085cf4fab4322e0636cc1c758c1a3716f00468b67a966135089b9bdefb141a6/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/7bbde9c26b246450bb31906007bafb324f31a112b451a601dae7c618060756db/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/2e7fd8f716566d264d06dda79414266b16e2fd045cde272a11d396e37e9acd2c/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/54f26bd21b58572eb6fc82e2b588118d56e1609df350ee6c6d3d619516f73996/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/250b67640d2045a2525a3b079db0674e83f3afdfbe12d7ef000077d3d9154beb/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/701e4637c1773707e5d21bab77b599569a8e35c1b8ce9f1d1512530a20c16e1a/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/e299552f934c301499fb752cdd3575731c4451f4ec198edcd3849d31786f3275/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/ee293e6f25364064b6d68101ecaac68408ccd23731d7d002cc9f2494364ebeec/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/65b25f0c54c239af75192899aeefa71c87eb3b38622c8b0e37396d99525642af/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7cb4caedfab050d86571ef89936b0e6c29bddecd744665dc49d4575d512dabf1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7c2f60f9cbcf4fd14765708d37cc0a7fd2271a6a37b06fd5c269dcfe128003f6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/fafe36bc47810a6f13cb60ea852a13e286fab8b89fed47784a6c474c04ad3d72/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/fa0276e97675c881fca4b3314e42f50b26f3296207f1df21ff210a7385cba2e0/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ea6a15df19742814a061a97d8868986c1a949a7d5593d4d3f8af37772ab93c66/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/b1a994603d3d39f884cf1baad95ad758018f7f4f714fa00aa55d044a01abc3d3/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/4ef3803e8137f2a0b808a005cfffa005d765bc58d4b8b66846d18ed9a07e3ebc/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/56255d4568be34da380af7532235c42f3e9d9668a6fe63262dad540af7cc0943/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 18:21:36.837630 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.837157    5551 manager.go:217] Machine: {Timestamp:2026-10-31 18:21:36.836626253 +0000 UTC m=+0.067864534 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3150718 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25912fb3436316d6440ea18661b548 SystemUUID:ec25912f-b343-6316-d644-0ea18661b548 BootID:c3512f76-0f79-4456-b4c5-0080dfff8e53 Filesystems:[{Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:242 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d24d184aa2ec9bcac439f2d949c9945598897bc7a9cfdaa271a427f0f703e74c/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:348 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fa123315d2dbc4848c9fc1f7961c421945e7dc165e4f3f8c1ccc2c2a1c8e467/userdata/shm DeviceMajor:0 DeviceMinor:413 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3386381a-8b41-473b-9400-04d3d3eb2c7e/volumes/kubernetes.io~projected/kube-api-access-mtb7n DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12b631c1-447d-43ee-9f5a-d77bfb04e03c/volumes/kubernetes.io~projected/kube-api-access-mpzgw DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec058e17-0198-4c81-8ce4-2e3a616f9d69/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:270 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:346 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3a917921a0754afb3d482b95d56520d071e371af584de5cc2278953ca1e216a/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aacf2191f4fd9790f551b62643322993358b8dd5805ed54198a9dda1ec8bc97d/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:353 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volumes/kubernetes.io~projected/kube-api-access-cprjg DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd649c93eab2f98b3997f01c56d2ff0f5c5385a2f6e0066591c4387d7dff8d5b/userdata/shm DeviceMajor:0 DeviceMinor:275 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:332 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~projected/kube-api-access-5fz2x DeviceMajor:0 DeviceMinor:412 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e2f392-7651-4d4d-9c09-b4978aabcebf/volumes/kubernetes.io~projected/kube-api-access-ldl6f DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69a446fa-5d09-4f26-9b5d-72f3fbe42064/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:196 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:404 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:411 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3dac68756a73a04f709f12b3d6c1eefa758c545abf17718f7b5a361f7e6cf4f4/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce0d6948e36ad007965b74f660246b4f836c2b0cd11ee3c75e7136b268d39bca/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/97e5d8ad-0765-4815-a495-53de9ca7b24b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:200 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:243 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:410 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac28dc82845f0baae359492484e9bb817e94a43c75317221fe99aa941d1c0a11/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1fe78dfe-d1f0-4196-9832-702532afcaf8/volumes/kubernetes.io~projected/kube-api-access-czvds DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:339 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:351 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:291 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:349 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:352 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a065905-b86e-4326-a7d8-ec336cb938c7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3497b74a-87c3-4f27-91b7-dac04680a875/volumes/kubernetes.io~projected/kube-api-access-6m5tp DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b7fff8965badd296b83c5c42e37cd3cbc19e1263f5aeb185d9c54612146838a/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~projected/kube-api-access-bp7jq DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~projected/kube-api-access-whmgd DeviceMajor:0 DeviceMinor:356 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:389 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec058e17-0198-4c81-8ce4-2e3a616f9d69/volumes/kubernetes.io~projected/kube-api-access-jkcqf DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/461b92ce-a67b-4dbe-924b-9e7c84f07a02/volumes/kubernetes.io~projected/kube-api-access-v62m6 DeviceMajor:0 DeviceMinor:244 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:396 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~projected/kube-api-access-7p4cn DeviceMajor:0 DeviceMinor:397 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a131e89a883509aa1436e62d02354c7b2a4801e70a820d12a4987a78ebd2fd61/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89b6ce0535b96eba824e61d9d6c38ee6d1969cbb71aed8545acdd294c558236d/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb570da9627300e528f6a81428e9d8b7a91eb8e7a2557f2b41f573c3ac8168d6/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea3ab49407dac02aad13cc91281751709100163ad3a967086ca9df9775e0b579/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ad94d5c-d364-4d68-8795-70af86c61c4e/volumes/kubernetes.io~projected/kube-api-access-8bbbk DeviceMajor:0 DeviceMinor:277 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59110ac180d05d3383f18e51c3d2ef56424f0c974f43cd1b6e21bbcd044c6e28/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f5039aa0259af9d5c470852b5c56ce454f9100a23979e263cb35a0ba4ec0b8d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~projected/kube-api-access-jzp8k DeviceMajor:0 DeviceMinor:296 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/69a446fa-5d09-4f26-9b5d-72f3fbe42064/volumes/kubernetes.io~projected/kube-api-access-8dchr DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/445c9865-e1dc-467f-bb6f-f29e8c6b7f68/volumes/kubernetes.io~projected/kube-api-access-sdx5n DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:345 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a00ba90066290cc95585207e999b86e0b3da4b2b3aa11f4afd5dce756983b2f7/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b7695f34b7a55d47f3bb55a9990a93090dfd794267f1557d28f2ff1aa26a225/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97e5d8ad-0765-4815-a495-53de9ca7b24b/volumes/kubernetes.io~projected/kube-api-access-zrltf DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f7cf5d-858f-4ae2-b53e-c64881398c3f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:355 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882/volumes/kubernetes.io~projected/kube-api-access-64wql DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:297 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/018a1eaae75ee0d8c6ef2cd2fabc62a6e253fc6f8a0ac41eeafe39536af77525/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83fe51c6-ee5f-4ccf-a934-6d42a27d475b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:295 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~projected/kube-api-access-v6bg2 DeviceMajor:0 DeviceMinor:298 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8be4d751-a0c8-4350-ac3f-8e12d3eca25c/volumes/kubernetes.io~projected/kube-api-access-mmw9b DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2105f6e721f592c0f6014006b097116a9857285c6406f46010196dd957d790f1/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/093b014626279e61080b83681b46d60c99de3ef65c3f45de8aad99340e5d5c23/userdata/shm DeviceMajor:0 DeviceMinor:280 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8df7b013ba992048504b55b5954e8649371266a700cbc03b849d4b0604a8d704/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:344 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/deca97c4-946e-4974-b30e-dd0595a94633/volumes/kubernetes.io~projected/kube-api-access-cpss5 DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cee41971d4a8889f0ee9c6c672b2e0710c00d4c16c44b98d31ff9d3deeac4cb9/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~projected/kube-api-access-bk88q DeviceMajor:0 DeviceMinor:347 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2efd8852-abb6-40d2-afda-40fe91bbd18f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28594e99-2b75-44a6-8922-315e6b680700/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:274 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0097d97e1ef0c7adeaeb69eb26f8808d654557ad46b60bbb3bdf47bf65d84532/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e99dc45621c676dae190438818cba7c89449d024ff30eb3ba5e322562f8399d/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb3b9ece-b002-4973-95ed-862cd33ca046/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:388 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/884384ea-01db-45f2-8c04-2163de1950ca/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:409 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59805563b901f5adace385924880f9fa196174aa1bf0c290417a8e1c1390cb1a/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b37353a-7cbf-478e-b7c5-83a71f6c1986/volumes/kubernetes.io~projected/kube-api-access-qrt52 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82cbbb05cb44ecbf8c2dc8e3117ed3bd8fca4cc4c15893a923eddacb4159dbb3/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65c82ec5-b112-4e22-9e89-25b3a69fe6a2/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:292 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:093b014626279e6 MacAddress:da:ab:2d:76:41:2f Speed:10000 Mtu:8901} {Name:3dac68756a73a04 MacAddress:1e:e5:39:47:e5:bd Speed:10000 Mtu:8901} {Name:59805563b901f5a MacAddress:3a:8f:1f:44:b1:b5 Speed:10000 Mtu:8901} {Name:8b7fff8965badd2 MacAddress:fe:e0:da:29:8c:f5 Speed:10000 Mtu:8901} {Name:8df7b013ba99204 MacAddress:72:f5:f4:80:f5:b0 Speed:10000 Mtu:8901} {Name:8fa123315d2dbc4 MacAddress:9e:93:da:3f:18:6a Speed:10000 Mtu:8901} {Name:a00ba90066290cc MacAddress:02:e0:74:2f:e2:79 Speed:10000 Mtu:8901} {Name:a131e89a883509a MacAddress:ce:4d:83:6d:5b:04 Speed:10000 Mtu:8901} {Name:aacf2191f4fd979 MacAddress:92:97:84:37:a9:c8 Speed:10000 Mtu:8901} {Name:ac28dc82845f0ba MacAddress:32:6e:4d:34:50:4e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:44:f0:bd:22:1f Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:d9:32:46:83:ab Speed:0 Mtu:8901} {Name:c3a917921a0754a MacAddress:96:f6:e0:65:d9:e1 Speed:10000 Mtu:8901} {Name:cee41971d4a8889 MacAddress:9e:fe:98:3b:5a:4e Speed:10000 Mtu:8901} {Name:d24d184aa2ec9bc MacAddress:ae:e3:2b:98:ae:6a Speed:10000 Mtu:8901} {Name:ea3ab49407dac02 MacAddress:12:d5:3c:a4:f1:34 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:44:f0:bd:22:1f Speed:0 Mtu:9001} {Name:fd649c93eab2f98 MacAddress:0a:ec:2b:e9:1c:3d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ee:24:81:17:35:10 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:b9:7d:e8:e3:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 18:21:36.837966 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.837952    5551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 18:21:36.838074 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.838042    5551 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 18:21:36.839065 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.838904    5551 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 18:21:36.839334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839197    5551 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-99.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 18:21:36.839421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839336    5551 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 18:21:36.839421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839345    5551 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 18:21:36.839421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839355    5551 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 18:21:36.839421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839373    5551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 18:21:36.839563 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839435    5551 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 18:21:36.839611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839601    5551 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 18:21:36.839708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839699    5551 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 18:21:36.839744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839733    5551 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 18:21:36.839744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839742    5551 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 18:21:36.839811 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839751    5551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 18:21:36.839811 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839759    5551 kubelet.go:405] "Adding apiserver pod source"
Oct 31 18:21:36.839811 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.839767    5551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 18:21:36.840437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.840421    5551 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 18:21:36.840437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.840434    5551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 18:21:36.840887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.840870    5551 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 18:21:36.856450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856434    5551 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 18:21:36.856450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856448    5551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 18:21:36.856574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856567    5551 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 18:21:36.856750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856739    5551 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 18:21:36.856780 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856768    5551 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 18:21:36.856868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856858    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 18:21:36.856905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856879    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 18:21:36.856905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856892    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 18:21:36.856905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856901    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856907    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856913    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856918    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856923    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856929    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856935    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856943    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856955    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856985    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 18:21:36.856999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.856995    5551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 18:21:36.860521 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.860507    5551 server.go:1263] "Started kubelet"
Oct 31 18:21:36.861101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.860559    5551 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 18:21:36.861101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.860633    5551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 18:21:36.861101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.860745    5551 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 18:21:36.861101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.861053    5551 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 18:21:36.861083 ip-10-0-132-99 systemd[1]: Started Kubernetes Kubelet.
Oct 31 18:21:36.862200 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.862109    5551 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 18:21:36.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.866837    5551 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 18:21:36.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.866859    5551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 18:21:36.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.866921    5551 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 18:21:36.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.866938    5551 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 18:21:36.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.866998    5551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:22 +0000 UTC" deadline="2028-05-21 09:28:54.326241451 +0000 UTC"
Oct 31 18:21:36.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.867017    5551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13623h7m17.459228106s"
Oct 31 18:21:36.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.867021    5551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 18:21:36.867310 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:36.867045    5551 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-99.us-west-1.compute.internal\" not found"
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.867633    5551 factory.go:55] Registering systemd factory
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.867649    5551 factory.go:223] Registration of the systemd container factory successfully
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.867996    5551 factory.go:153] Registering CRI-O factory
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.868008    5551 factory.go:223] Registration of the crio container factory successfully
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.868051    5551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.868068    5551 factory.go:103] Registering Raw factory
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.868082    5551 manager.go:1199] Started watching for new ooms in manager
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.868397    5551 manager.go:319] Starting recovery of all containers
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.870925    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.870976    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a065905-b86e-4326-a7d8-ec336cb938c7" volumeName="kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.870989    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3386381a-8b41-473b-9400-04d3d3eb2c7e" volumeName="kubernetes.io/projected/3386381a-8b41-473b-9400-04d3d3eb2c7e-kube-api-access-mtb7n" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871013    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="519485a6-66ad-46f3-b5bc-76e5bc68315e" volumeName="kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871026    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69a446fa-5d09-4f26-9b5d-72f3fbe42064" volumeName="kubernetes.io/secret/69a446fa-5d09-4f26-9b5d-72f3fbe42064-metrics-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871037    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3b9ece-b002-4973-95ed-862cd33ca046" volumeName="kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871050    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882" volumeName="kubernetes.io/projected/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-kube-api-access-64wql" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871060    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe78dfe-d1f0-4196-9832-702532afcaf8" volumeName="kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-cni-binary-copy" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871073    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="445c9865-e1dc-467f-bb6f-f29e8c6b7f68" volumeName="kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871109    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c82ec5-b112-4e22-9e89-25b3a69fe6a2" volumeName="kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871121    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c82ec5-b112-4e22-9e89-25b3a69fe6a2" volumeName="kubernetes.io/projected/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-api-access-v6bg2" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871142    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69a446fa-5d09-4f26-9b5d-72f3fbe42064" volumeName="kubernetes.io/projected/69a446fa-5d09-4f26-9b5d-72f3fbe42064-kube-api-access-8dchr" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871154    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871165    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec058e17-0198-4c81-8ce4-2e3a616f9d69" volumeName="kubernetes.io/projected/ec058e17-0198-4c81-8ce4-2e3a616f9d69-kube-api-access-jkcqf" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871179    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe78dfe-d1f0-4196-9832-702532afcaf8" volumeName="kubernetes.io/projected/1fe78dfe-d1f0-4196-9832-702532afcaf8-kube-api-access-czvds" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871203    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2efd8852-abb6-40d2-afda-40fe91bbd18f" volumeName="kubernetes.io/configmap/2efd8852-abb6-40d2-afda-40fe91bbd18f-konnectivity-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871214    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3386381a-8b41-473b-9400-04d3d3eb2c7e" volumeName="kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-binary-copy" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871226    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deca97c4-946e-4974-b30e-dd0595a94633" volumeName="kubernetes.io/empty-dir/deca97c4-946e-4974-b30e-dd0595a94633-tmp" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871237    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871248    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3386381a-8b41-473b-9400-04d3d3eb2c7e" volumeName="kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871259    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c82ec5-b112-4e22-9e89-25b3a69fe6a2" volumeName="kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-metrics-client-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871269    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97e5d8ad-0765-4815-a495-53de9ca7b24b" volumeName="kubernetes.io/empty-dir/97e5d8ad-0765-4815-a495-53de9ca7b24b-data-volume" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871282    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882" volumeName="kubernetes.io/secret/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-cert" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871292    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a065905-b86e-4326-a7d8-ec336cb938c7" volumeName="kubernetes.io/secret/0a065905-b86e-4326-a7d8-ec336cb938c7-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871305    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871327    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871338    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884384ea-01db-45f2-8c04-2163de1950ca" volumeName="kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871349    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871366    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deca97c4-946e-4974-b30e-dd0595a94633" volumeName="kubernetes.io/projected/deca97c4-946e-4974-b30e-dd0595a94633-kube-api-access-cpss5" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871379    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2efd8852-abb6-40d2-afda-40fe91bbd18f" volumeName="kubernetes.io/secret/2efd8852-abb6-40d2-afda-40fe91bbd18f-agent-certs" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871396    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3386381a-8b41-473b-9400-04d3d3eb2c7e" volumeName="kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871407    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c82ec5-b112-4e22-9e89-25b3a69fe6a2" volumeName="kubernetes.io/empty-dir/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-volume-directive-shadow" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871423    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fe51c6-ee5f-4ccf-a934-6d42a27d475b" volumeName="kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-metrics-client-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871470    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8be4d751-a0c8-4350-ac3f-8e12d3eca25c" volumeName="kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-tmp" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871483    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97e5d8ad-0765-4815-a495-53de9ca7b24b" volumeName="kubernetes.io/configmap/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871494    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e2f392-7651-4d4d-9c09-b4978aabcebf" volumeName="kubernetes.io/projected/a8e2f392-7651-4d4d-9c09-b4978aabcebf-kube-api-access-ldl6f" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871519    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/configmap/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-metrics-client-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871542    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/empty-dir/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-tmp" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871583    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3b9ece-b002-4973-95ed-862cd33ca046" volumeName="kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871608    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871648    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884384ea-01db-45f2-8c04-2163de1950ca" volumeName="kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871673    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884384ea-01db-45f2-8c04-2163de1950ca" volumeName="kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-client-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871699    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3b9ece-b002-4973-95ed-862cd33ca046" volumeName="kubernetes.io/projected/cb3b9ece-b002-4973-95ed-862cd33ca046-kube-api-access-7p4cn" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871712    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec058e17-0198-4c81-8ce4-2e3a616f9d69" volumeName="kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871723    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="519485a6-66ad-46f3-b5bc-76e5bc68315e" volumeName="kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871745    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884384ea-01db-45f2-8c04-2163de1950ca" volumeName="kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-metrics-client-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871756    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884384ea-01db-45f2-8c04-2163de1950ca" volumeName="kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871767    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884384ea-01db-45f2-8c04-2163de1950ca" volumeName="kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-federate-client-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871778    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3b9ece-b002-4973-95ed-862cd33ca046" volumeName="kubernetes.io/empty-dir/cb3b9ece-b002-4973-95ed-862cd33ca046-audit-log" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871790    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871802    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3497b74a-87c3-4f27-91b7-dac04680a875" volumeName="kubernetes.io/projected/3497b74a-87c3-4f27-91b7-dac04680a875-kube-api-access-6m5tp" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871895    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="445c9865-e1dc-467f-bb6f-f29e8c6b7f68" volumeName="kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871914    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="461b92ce-a67b-4dbe-924b-9e7c84f07a02" volumeName="kubernetes.io/projected/461b92ce-a67b-4dbe-924b-9e7c84f07a02-kube-api-access-v62m6" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871927    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="461b92ce-a67b-4dbe-924b-9e7c84f07a02" volumeName="kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871939    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="519485a6-66ad-46f3-b5bc-76e5bc68315e" volumeName="kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871952    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69a446fa-5d09-4f26-9b5d-72f3fbe42064" volumeName="kubernetes.io/empty-dir/69a446fa-5d09-4f26-9b5d-72f3fbe42064-tmp-dir" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871973    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8be4d751-a0c8-4350-ac3f-8e12d3eca25c" volumeName="kubernetes.io/projected/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-kube-api-access-mmw9b" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871986    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a065905-b86e-4326-a7d8-ec336cb938c7" volumeName="kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-env-overrides" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.871996    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872011    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872022    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872032    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="519485a6-66ad-46f3-b5bc-76e5bc68315e" volumeName="kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872043    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8be4d751-a0c8-4350-ac3f-8e12d3eca25c" volumeName="kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-tuned" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872053    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e2f392-7651-4d4d-9c09-b4978aabcebf" volumeName="kubernetes.io/configmap/a8e2f392-7651-4d4d-9c09-b4978aabcebf-serviceca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872064    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872076    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b631c1-447d-43ee-9f5a-d77bfb04e03c" volumeName="kubernetes.io/projected/12b631c1-447d-43ee-9f5a-d77bfb04e03c-kube-api-access-mpzgw" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872089    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="445c9865-e1dc-467f-bb6f-f29e8c6b7f68" volumeName="kubernetes.io/configmap/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-metrics-client-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872100    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="461b92ce-a67b-4dbe-924b-9e7c84f07a02" volumeName="kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872110    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="519485a6-66ad-46f3-b5bc-76e5bc68315e" volumeName="kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872124    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c82ec5-b112-4e22-9e89-25b3a69fe6a2" volumeName="kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872134    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fe51c6-ee5f-4ccf-a934-6d42a27d475b" volumeName="kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872145    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872155    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-grpc-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872170    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b631c1-447d-43ee-9f5a-d77bfb04e03c" volumeName="kubernetes.io/empty-dir/12b631c1-447d-43ee-9f5a-d77bfb04e03c-tmp-dir" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872179    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28594e99-2b75-44a6-8922-315e6b680700" volumeName="kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872186    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872193    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ad94d5c-d364-4d68-8795-70af86c61c4e" volumeName="kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872202    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="519485a6-66ad-46f3-b5bc-76e5bc68315e" volumeName="kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872212    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3b9ece-b002-4973-95ed-862cd33ca046" volumeName="kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872237    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3b9ece-b002-4973-95ed-862cd33ca046" volumeName="kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-client-ca-bundle" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872248    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3b9ece-b002-4973-95ed-862cd33ca046" volumeName="kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872256    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872264    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872271    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872279    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3497b74a-87c3-4f27-91b7-dac04680a875" volumeName="kubernetes.io/configmap/3497b74a-87c3-4f27-91b7-dac04680a875-iptables-alerter-script" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872289    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="445c9865-e1dc-467f-bb6f-f29e8c6b7f68" volumeName="kubernetes.io/projected/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-kube-api-access-sdx5n" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872300    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="461b92ce-a67b-4dbe-924b-9e7c84f07a02" volumeName="kubernetes.io/configmap/461b92ce-a67b-4dbe-924b-9e7c84f07a02-metrics-client-ca" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872312    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c82ec5-b112-4e22-9e89-25b3a69fe6a2" volumeName="kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872325    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69a446fa-5d09-4f26-9b5d-72f3fbe42064" volumeName="kubernetes.io/configmap/69a446fa-5d09-4f26-9b5d-72f3fbe42064-config-volume" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872336    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe78dfe-d1f0-4196-9832-702532afcaf8" volumeName="kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-daemon-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872350    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fe51c6-ee5f-4ccf-a934-6d42a27d475b" volumeName="kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872361    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fe51c6-ee5f-4ccf-a934-6d42a27d475b" volumeName="kubernetes.io/projected/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-kube-api-access-jzp8k" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872368    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fe51c6-ee5f-4ccf-a934-6d42a27d475b" volumeName="kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872375    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884384ea-01db-45f2-8c04-2163de1950ca" volumeName="kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872385    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884384ea-01db-45f2-8c04-2163de1950ca" volumeName="kubernetes.io/projected/884384ea-01db-45f2-8c04-2163de1950ca-kube-api-access-5fz2x" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872394    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97e5d8ad-0765-4815-a495-53de9ca7b24b" volumeName="kubernetes.io/projected/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-api-access-zrltf" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872403    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97e5d8ad-0765-4815-a495-53de9ca7b24b" volumeName="kubernetes.io/secret/97e5d8ad-0765-4815-a495-53de9ca7b24b-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872412    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a065905-b86e-4326-a7d8-ec336cb938c7" volumeName="kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-config" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872424    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a065905-b86e-4326-a7d8-ec336cb938c7" volumeName="kubernetes.io/projected/0a065905-b86e-4326-a7d8-ec336cb938c7-kube-api-access-cprjg" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872436    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c130b20-c7b5-46a5-8c19-59ca36d172c8" volumeName="kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872448    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="519485a6-66ad-46f3-b5bc-76e5bc68315e" volumeName="kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872487    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83fe51c6-ee5f-4ccf-a934-6d42a27d475b" volumeName="kubernetes.io/empty-dir/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-textfile" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872500    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b37353a-7cbf-478e-b7c5-83a71f6c1986" volumeName="kubernetes.io/projected/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kube-api-access-qrt52" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872512    5551 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7cf5d-858f-4ae2-b53e-c64881398c3f" volumeName="kubernetes.io/projected/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-kube-api-access-whmgd" seLinuxMountContext=""
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872523    5551 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 18:21:36.879112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.872531    5551 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 18:21:36.899355 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.899326    5551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 18:21:36.900954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.900936    5551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 18:21:36.900954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.900957    5551 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 18:21:36.901091 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.900973    5551 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 18:21:36.901091 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:36.901005    5551 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 18:21:36.923629 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.923599    5551 generic.go:356] "Generic (PLEG): container finished" podID="adc1d611432cbd40b2e3811e0cea4a6f" containerID="b0f7a6b276247df54ed639c4d6df9fc9843e3cd68f7f41f3ec14379a7d29d169" exitCode=0
Oct 31 18:21:36.936704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.936640    5551 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="0ce66beea8f3997cd7f3db2e0b2d59534e1140835daecd790fcc5a4d508c1765" exitCode=0
Oct 31 18:21:36.936704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.936668    5551 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="da5e38825292c592f1a61ec65f6f43d3f1afa2dd99c37654cad4e6efb5ef2017" exitCode=0
Oct 31 18:21:36.936704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.936679    5551 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="dd1574ca726778db56205c8ddf96ca4e9b65bf2e7aa7f26c966221853a1f2cc3" exitCode=0
Oct 31 18:21:36.936704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.936688    5551 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="3725d11fa5414dfc393507b2231ba4ad2a4592cb5f98fcf3590d721f44329391" exitCode=0
Oct 31 18:21:36.936704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.936696    5551 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="baffe93be09bea7630e8fbbd7fc6016f1af5cda17c9f012213d0b7eabfcdb6bd" exitCode=0
Oct 31 18:21:36.936704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.936704    5551 generic.go:356] "Generic (PLEG): container finished" podID="3386381a-8b41-473b-9400-04d3d3eb2c7e" containerID="b2c02bc3807c57f874c00d27f61048ea799e6742e181e61da6b922677de168e0" exitCode=0
Oct 31 18:21:36.945493 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.945449    5551 generic.go:356] "Generic (PLEG): container finished" podID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerID="23fb516d57f49fe77b6344c8f6d890a0ecdd2ac64f904ad7d452fedfcfb8c089" exitCode=0
Oct 31 18:21:36.967173 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:36.967145    5551 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-99.us-west-1.compute.internal\" not found"
Oct 31 18:21:36.982559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.982518    5551 generic.go:356] "Generic (PLEG): container finished" podID="83fe51c6-ee5f-4ccf-a934-6d42a27d475b" containerID="5c52af85dabc9e2725ce1a088597e26758994da56c1eed5cdf4f53438555c300" exitCode=0
Oct 31 18:21:36.994553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:36.992893    5551 manager.go:324] Recovery completed
Oct 31 18:21:37.001594 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:37.001564    5551 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 18:21:37.017679 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.017654    5551 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:21:37.019549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.019530    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:21:37.019653 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.019562    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:21:37.019653 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.019573    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.022801    5551 cpu_manager.go:226] "Starting" policy="none"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.022818    5551 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.022837    5551 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.022987    5551 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023008    5551 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023030    5551 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023037    5551 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023045    5551 policy_none.go:50] "Start"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023055    5551 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023066    5551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023227    5551 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023237    5551 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 18:21:37.023868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.023245    5551 policy_none.go:44] "Start"
Oct 31 18:21:37.025674 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.025655    5551 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:21:37.026910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.026884    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:21:37.027019 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.026925    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:21:37.027019 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.026941    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:21:37.031153 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.031116    5551 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 18:21:37.035549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.035534    5551 manager.go:341] "Starting Device Plugin manager"
Oct 31 18:21:37.035656 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:37.035564    5551 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 18:21:37.035656 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.035578    5551 server.go:87] "Starting device plugin registration server"
Oct 31 18:21:37.035841 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.035828    5551 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 18:21:37.035903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.035841    5551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 18:21:37.036008 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.035997    5551 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 18:21:37.036103 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.036072    5551 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 18:21:37.036163 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.036108    5551 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 18:21:37.036163 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.036113    5551 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 18:21:37.036982 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.036171    5551 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T18:21:37.036100528Z","UUID":"506eb402-848f-4f2e-b7b0-f0445d71fd61","Handler":null,"Name":"","Endpoint":""}
Oct 31 18:21:37.038303 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:37.038279    5551 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 18:21:37.038420 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:37.038325    5551 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.us-west-1.compute.internal\" not found"
Oct 31 18:21:37.038529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.038489    5551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 18:21:37.038529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.038514    5551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 18:21:37.048440 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.048422    5551 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:21:37.048820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.048803    5551 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:21:37.048975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.048961    5551 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 18:21:37.049113 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.049094    5551 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:21:37.136802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.136784    5551 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:21:37.137739 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.137723    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:21:37.137810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.137752    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:21:37.137810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.137763    5551 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-99.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:21:37.137810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.137807    5551 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.147638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.147612    5551 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:21:37.202293 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.202213    5551 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"]
Oct 31 18:21:37.202882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.202848    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"adc1d611432cbd40b2e3811e0cea4a6f","Type":"ContainerStarted","Data":"fe2294365af9d121bef3ddec65b8ea1018f93fdbd1145f1a87508dbcfad17737"}
Oct 31 18:21:37.202975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.202891    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"adc1d611432cbd40b2e3811e0cea4a6f","Type":"ContainerDied","Data":"b0f7a6b276247df54ed639c4d6df9fc9843e3cd68f7f41f3ec14379a7d29d169"}
Oct 31 18:21:37.202975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.202912    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"adc1d611432cbd40b2e3811e0cea4a6f","Type":"ContainerStarted","Data":"2105f6e721f592c0f6014006b097116a9857285c6406f46010196dd957d790f1"}
Oct 31 18:21:37.202975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.202966    5551 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.203119 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.202968    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"401f601c1d615f3dbfc663ac1fbee6e2","Type":"ContainerStarted","Data":"526dbadcd0b4f7e718eabfbb46a8fc61a0a17cbba4ddfc804b7df0d015eff5e6"}
Oct 31 18:21:37.203119 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.203048    5551 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.203119 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.203064    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal" event={"ID":"401f601c1d615f3dbfc663ac1fbee6e2","Type":"ContainerStarted","Data":"59110ac180d05d3383f18e51c3d2ef56424f0c974f43cd1b6e21bbcd044c6e28"}
Oct 31 18:21:37.252538 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.252522    5551 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.252648 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.252577    5551 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.263983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.263962    5551 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:21:37.264052 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:37.264002    5551 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.264408 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.264392    5551 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:21:37.264501 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:37.264433    5551 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.350611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.350581    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/401f601c1d615f3dbfc663ac1fbee6e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"401f601c1d615f3dbfc663ac1fbee6e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.350742 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.350635    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.350742 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.350659    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.451209 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.451172    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/401f601c1d615f3dbfc663ac1fbee6e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"401f601c1d615f3dbfc663ac1fbee6e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.451209 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.451192    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/401f601c1d615f3dbfc663ac1fbee6e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"401f601c1d615f3dbfc663ac1fbee6e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.451396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.451224    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.451396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.451248    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.451396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.451289    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.451396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.451321    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/adc1d611432cbd40b2e3811e0cea4a6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" (UID: \"adc1d611432cbd40b2e3811e0cea4a6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.840429 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.840399    5551 apiserver.go:52] "Watching apiserver"
Oct 31 18:21:37.946570 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.946534    5551 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 18:21:37.949999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.949970    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-695c5cddbb-m2rvq","openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n","openshift-image-registry/node-ca-rlwgx","openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5","openshift-monitoring/metrics-server-7ff57c786f-q298q","openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4","openshift-network-diagnostics/network-check-target-ztv9x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc","openshift-dns/dns-default-8z5vc","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-bx85h","openshift-multus/network-metrics-daemon-mgb86","kube-system/global-pull-secret-syncer-m56gf","openshift-cluster-node-tuning-operator/tuned-4g2hb","openshift-insights/insights-runtime-extractor-g8rpt","kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-rg4dh","openshift-monitoring/node-exporter-fgkm2","openshift-console/downloads-6dd789c8b7-8cp96","openshift-network-operator/iptables-alerter-25sgx","openshift-ovn-kubernetes/ovnkube-node-qtzdz","kube-system/konnectivity-agent-hgxrq","openshift-console/console-7986dc697c-f54gg","openshift-dns/node-resolver-xjpst","openshift-ingress-canary/ingress-canary-zkb25","openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"]
Oct 31 18:21:37.970130 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.970091    5551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 18:21:37.990245 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.990220    5551 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:37.990405 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:37.990343    5551 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:38.007150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.007112    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.007568 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.007539    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 18:21:38.007685 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.007590    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.008956 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.008929    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 18:21:38.009074 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009060    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72cf5\""
Oct 31 18:21:38.009196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009179    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7mm2n\""
Oct 31 18:21:38.009292 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009278    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.009367 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009302    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-szhfk\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009412    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009515    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m5pkq\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009553    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgxxf\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009792    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.009824    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010028    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010071    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xr757\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010138    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vv64h\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010196    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010224    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010378    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 18:21:38.010450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010449    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9xfhz\""
Oct 31 18:21:38.010998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010505    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.010998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010510    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 18:21:38.010998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010602    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.010998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010635    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 18:21:38.010998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010704    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.010998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010769    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.010998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.010942    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 18:21:38.012412 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.012393    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 18:21:38.054344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054316    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtb7n\" (UniqueName: \"kubernetes.io/projected/3386381a-8b41-473b-9400-04d3d3eb2c7e-kube-api-access-mtb7n\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.054344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054346    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4cn\" (UniqueName: \"kubernetes.io/projected/cb3b9ece-b002-4973-95ed-862cd33ca046-kube-api-access-7p4cn\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.054559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054367    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.054559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054383    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-tls\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.054559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054403    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/a8e2f392-7651-4d4d-9c09-b4978aabcebf-kube-api-access-ldl6f\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:21:38.054559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054431    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-sys-fs\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.054559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054474    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fz2x\" (UniqueName: \"kubernetes.io/projected/884384ea-01db-45f2-8c04-2163de1950ca-kube-api-access-5fz2x\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.054559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054501    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-tuned\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.054559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054530    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-sys\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.054559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054556    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054575    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-env-overrides\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054597    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-grpc-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054614    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-tuned\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054621    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2efd8852-abb6-40d2-afda-40fe91bbd18f-agent-certs\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054692    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-etc-selinux\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054741    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-cert\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054774    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054801    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-multus-certs\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054828    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czvds\" (UniqueName: \"kubernetes.io/projected/1fe78dfe-d1f0-4196-9832-702532afcaf8-kube-api-access-czvds\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054852    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054854    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-env-overrides\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054877    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2efd8852-abb6-40d2-afda-40fe91bbd18f-agent-certs\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:21:38.054980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.054885    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-var-lib-kubelet\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055008    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97e5d8ad-0765-4815-a495-53de9ca7b24b-crio-socket\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055041    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97e5d8ad-0765-4815-a495-53de9ca7b24b-data-volume\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055059    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-etc-kubernetes\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055116    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-client-certs\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055138    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97e5d8ad-0765-4815-a495-53de9ca7b24b-data-volume\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055141    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055173    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055204    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055231    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055257    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-federate-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055331    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-ovn\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055357    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-log-socket\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055383    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a446fa-5d09-4f26-9b5d-72f3fbe42064-tmp-dir\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055415    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dchr\" (UniqueName: \"kubernetes.io/projected/69a446fa-5d09-4f26-9b5d-72f3fbe42064-kube-api-access-8dchr\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055502    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055526    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a446fa-5d09-4f26-9b5d-72f3fbe42064-tmp-dir\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055566    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-systemd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055604    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpss5\" (UniqueName: \"kubernetes.io/projected/deca97c4-946e-4974-b30e-dd0595a94633-kube-api-access-cpss5\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055632    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b631c1-447d-43ee-9f5a-d77bfb04e03c-tmp-dir\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055658    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3497b74a-87c3-4f27-91b7-dac04680a875-iptables-alerter-script\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055674    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.055693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055697    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-config\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055721    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-sys\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055738    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b631c1-447d-43ee-9f5a-d77bfb04e03c-tmp-dir\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055799    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055823    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/461b92ce-a67b-4dbe-924b-9e7c84f07a02-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055842    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-netns\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055870    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-system-cni-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055919    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62m6\" (UniqueName: \"kubernetes.io/projected/461b92ce-a67b-4dbe-924b-9e7c84f07a02-kube-api-access-v62m6\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055969    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-config\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.055975    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a446fa-5d09-4f26-9b5d-72f3fbe42064-metrics-tls\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056011    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-tmp\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056038    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056064    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-serving-certs-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056071    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-tmp\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056092    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-kubelet-config\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056117    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056146    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcqf\" (UniqueName: \"kubernetes.io/projected/ec058e17-0198-4c81-8ce4-2e3a616f9d69-kube-api-access-jkcqf\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056166    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-tls\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056191    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-registration-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056208    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-system-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056256    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-multus\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056286    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056303    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-metrics-client-ca\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056332    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056357    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12b631c1-447d-43ee-9f5a-d77bfb04e03c-hosts-file\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056373    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056389    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-systemd\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056432    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-bin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056449    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-conf\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056490    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056507    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-script-lib\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056523    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-root\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056537    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deca97c4-946e-4974-b30e-dd0595a94633-tmp\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056555    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056627    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deca97c4-946e-4974-b30e-dd0595a94633-tmp\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056662    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056700    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-systemd-units\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056729    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-wtmp\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056760    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cb3b9ece-b002-4973-95ed-862cd33ca046-audit-log\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056765    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a065905-b86e-4326-a7d8-ec336cb938c7-ovnkube-script-lib\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056787    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.056775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056808    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056825    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-modprobe-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056840    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056848    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cb3b9ece-b002-4973-95ed-862cd33ca046-audit-log\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056862    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056925    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056955    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.056983    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057014    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057041    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057068    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057042    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057093    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-socket-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057125    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-etc-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057153    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97e5d8ad-0765-4815-a495-53de9ca7b24b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057176    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzp8k\" (UniqueName: \"kubernetes.io/projected/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-kube-api-access-jzp8k\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057193    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057232    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-cni-binary-copy\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057309    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057330    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057373    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-kubelet\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057401    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057425    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrltf\" (UniqueName: \"kubernetes.io/projected/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-api-access-zrltf\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057442    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-metrics-client-ca\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057473    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8e2f392-7651-4d4d-9c09-b4978aabcebf-serviceca\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057527    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-k8s-cni-cncf-io\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057553    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-kubernetes\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057599    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-host\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057625    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-socket-dir-parent\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057671    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7jq\" (UniqueName: \"kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057697    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057718    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057733    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-cnibin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057754    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057778    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057806    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057830    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-metrics-server-audit-profiles\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057857    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64wql\" (UniqueName: \"kubernetes.io/projected/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-kube-api-access-64wql\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057869    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057878    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzgw\" (UniqueName: \"kubernetes.io/projected/12b631c1-447d-43ee-9f5a-d77bfb04e03c-kube-api-access-mpzgw\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057899    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a065905-b86e-4326-a7d8-ec336cb938c7-ovn-node-metrics-cert\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057926    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-textfile\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057934    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057950    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.057973    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2efd8852-abb6-40d2-afda-40fe91bbd18f-konnectivity-ca\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058003    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058027    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-textfile\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058032    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-run\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058073    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28594e99-2b75-44a6-8922-315e6b680700-original-pull-secret\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058094    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058130    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8e2f392-7651-4d4d-9c09-b4978aabcebf-serviceca\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058189    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2efd8852-abb6-40d2-afda-40fe91bbd18f-konnectivity-ca\") pod \"konnectivity-agent-hgxrq\" (UID: \"2efd8852-abb6-40d2-afda-40fe91bbd18f\") " pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058221    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-daemon-config\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058243    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx5n\" (UniqueName: \"kubernetes.io/projected/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-kube-api-access-sdx5n\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058272    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a065905-b86e-4326-a7d8-ec336cb938c7-ovn-node-metrics-cert\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058281    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058307    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-dbus\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058333    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysconfig\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058351    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-lib-modules\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058350    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058391    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-netns\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058423    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-metrics-client-ca\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058473    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058550    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-hostroot\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058583    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-node-log\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058610    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058638    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-device-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058659    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6bg2\" (UniqueName: \"kubernetes.io/projected/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-api-access-v6bg2\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058690    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-tmp\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058706    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-bin\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.058708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058729    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058771    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058797    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058813    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058827    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-slash\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058842    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-netd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058857    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a446fa-5d09-4f26-9b5d-72f3fbe42064-config-volume\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058905    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5tp\" (UniqueName: \"kubernetes.io/projected/3497b74a-87c3-4f27-91b7-dac04680a875-kube-api-access-6m5tp\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058922    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058944    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058961    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058977    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk88q\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.058992    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-cnibin\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059013    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprjg\" (UniqueName: \"kubernetes.io/projected/0a065905-b86e-4326-a7d8-ec336cb938c7-kube-api-access-cprjg\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059034    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-client-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059049    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3497b74a-87c3-4f27-91b7-dac04680a875-host-slash\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059064    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059132    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmgd\" (UniqueName: \"kubernetes.io/projected/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-kube-api-access-whmgd\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059148    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-kubelet\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059166    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059183    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmw9b\" (UniqueName: \"kubernetes.io/projected/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-kube-api-access-mmw9b\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059199    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-var-lib-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059214    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e2f392-7651-4d4d-9c09-b4978aabcebf-host\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059231    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrt52\" (UniqueName: \"kubernetes.io/projected/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kube-api-access-qrt52\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059246    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-conf-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059262    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.059775    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-tmp\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.060844    5551 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.060870    5551 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:38.060901    5551 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:38.061580 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:38.061120    5551 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.us-west-1.compute.internal"
Oct 31 18:21:38.106054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.105998    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.106224 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.106211    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.106877 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.106858    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.107099 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.107084    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.107252 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.107230    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 18:21:38.107343 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.107329    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 18:21:38.107413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.107381    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 18:21:38.107575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.107550    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 18:21:38.107673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.107652    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 18:21:38.108005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.107985    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 18:21:38.108123 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108109    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 18:21:38.108236 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108217    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.108300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108272    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-b8zc6\""
Oct 31 18:21:38.108300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108288    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.108371 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108290    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.108413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108374    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 18:21:38.108512 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108498    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.108553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108546    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 18:21:38.108597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108561    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 18:21:38.108597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108572    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.108597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108562    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 18:21:38.108711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108694    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.108864 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108848    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 18:21:38.108939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.108923    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 18:21:38.109054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109039    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 18:21:38.109195 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109183    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 18:21:38.109195 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109191    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 18:21:38.109297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109221    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 18:21:38.109297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109234    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 18:21:38.109297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109242    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 18:21:38.109297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109251    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 18:21:38.109297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109280    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 18:21:38.109297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109285    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 18:21:38.109297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109295    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109301    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109302    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109349    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109355    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109405    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lvpkw\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109482    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109536    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109545    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109546    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109571    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109587    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109592    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.109607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109588    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.110045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109626    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.110045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109671    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 18:21:38.110045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109732    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 18:21:38.110045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109808    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4kdr\""
Oct 31 18:21:38.110045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.109985    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.110045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110028    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110062    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fo5e1905l2a1u\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110062    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110076    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110100    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qqqsk\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110109    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110126    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l6bdm\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110187    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fk2vk\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110191    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110207    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110224    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110243    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110257    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110290    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110308    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 18:21:38.110314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110311    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms9mk\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110344    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110359    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110379    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110416    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110432    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bm89f\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110525    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110556    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9l6d9\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110585    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w82k4\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110589    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bcwr9\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110613    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110619    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-97cx7\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110642    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110666    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dj9hm\""
Oct 31 18:21:38.110849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110679    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkjx2\""
Oct 31 18:21:38.111280 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110914    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-27pxv\""
Oct 31 18:21:38.111280 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.110916    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-77xvx\""
Oct 31 18:21:38.111280 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.111176    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 18:21:38.111280 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.111184    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 18:21:38.111434 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.111296    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5t2xq\""
Oct 31 18:21:38.111434 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.111301    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 18:21:38.111434 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.111309    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 18:21:38.111591 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.111494    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1eblk8drb2j08\""
Oct 31 18:21:38.112181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.112161    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/a8e2f392-7651-4d4d-9c09-b4978aabcebf-kube-api-access-ldl6f\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:21:38.115229 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115193    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-cert\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:38.115229 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115192    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.115353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115238    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.115353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115253    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-tls\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.115353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115256    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.115353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115253    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.115353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115277    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-grpc-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.115353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115334    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.115553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115407    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.115553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115426    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-secret-metrics-server-client-certs\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.115553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115478    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-secret-telemeter-client\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.115648 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115561    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/884384ea-01db-45f2-8c04-2163de1950ca-federate-client-tls\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.115678 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.115664    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.116592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116475    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-tls\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.116592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116489    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.116592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116504    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3497b74a-87c3-4f27-91b7-dac04680a875-iptables-alerter-script\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:21:38.116592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116511    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.116592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116520    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/461b92ce-a67b-4dbe-924b-9e7c84f07a02-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:38.116592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116546    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-serving-certs-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.116592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116553    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:38.116592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116563    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:38.116885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116653    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a446fa-5d09-4f26-9b5d-72f3fbe42064-metrics-tls\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:38.116885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116655    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-metrics-client-ca\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.116885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116676    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.116885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116696    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.116885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116751    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.116885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116852    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:38.117060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.116962    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.117092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117065    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.117092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117084    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.117242 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117225    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.117277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117236    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.117277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117263    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-tls\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.117277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117268    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.117383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117372    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97e5d8ad-0765-4815-a495-53de9ca7b24b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.117548 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117526    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.117610 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117533    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/461b92ce-a67b-4dbe-924b-9e7c84f07a02-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:38.117610 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117554    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3386381a-8b41-473b-9400-04d3d3eb2c7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.117610 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117564    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-cni-binary-copy\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.117701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117687    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058e17-0198-4c81-8ce4-2e3a616f9d69-metrics-certs\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:38.117730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.117708    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-metrics-client-ca\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.118906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.118887    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.118975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.118905    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.118975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.118920    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-daemon-config\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.118975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.118909    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.118975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.118944    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-metrics-client-ca\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.119096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.119081    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cb3b9ece-b002-4973-95ed-862cd33ca046-metrics-server-audit-profiles\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.120150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.120137    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a446fa-5d09-4f26-9b5d-72f3fbe42064-config-volume\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:38.120185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.120138    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cb3b9ece-b002-4973-95ed-862cd33ca046-client-ca-bundle\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.123197 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.123169    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcqf\" (UniqueName: \"kubernetes.io/projected/ec058e17-0198-4c81-8ce4-2e3a616f9d69-kube-api-access-jkcqf\") pod \"network-metrics-daemon-mgb86\" (UID: \"ec058e17-0198-4c81-8ce4-2e3a616f9d69\") " pod="openshift-multus/network-metrics-daemon-mgb86"
Oct 31 18:21:38.123291 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.123170    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtb7n\" (UniqueName: \"kubernetes.io/projected/3386381a-8b41-473b-9400-04d3d3eb2c7e-kube-api-access-mtb7n\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.123350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.123303    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fz2x\" (UniqueName: \"kubernetes.io/projected/884384ea-01db-45f2-8c04-2163de1950ca-kube-api-access-5fz2x\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.123350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.123327    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4cn\" (UniqueName: \"kubernetes.io/projected/cb3b9ece-b002-4973-95ed-862cd33ca046-kube-api-access-7p4cn\") pod \"metrics-server-7ff57c786f-q298q\" (UID: \"cb3b9ece-b002-4973-95ed-862cd33ca046\") " pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:38.123428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.123393    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62m6\" (UniqueName: \"kubernetes.io/projected/461b92ce-a67b-4dbe-924b-9e7c84f07a02-kube-api-access-v62m6\") pod \"prometheus-operator-9cd89b74c-wzh2n\" (UID: \"461b92ce-a67b-4dbe-924b-9e7c84f07a02\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wzh2n"
Oct 31 18:21:38.123548 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.123533    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpss5\" (UniqueName: \"kubernetes.io/projected/deca97c4-946e-4974-b30e-dd0595a94633-kube-api-access-cpss5\") pod \"downloads-6dd789c8b7-8cp96\" (UID: \"deca97c4-946e-4974-b30e-dd0595a94633\") " pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:21:38.123582 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.123167    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czvds\" (UniqueName: \"kubernetes.io/projected/1fe78dfe-d1f0-4196-9832-702532afcaf8-kube-api-access-czvds\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.157612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.157592    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dchr\" (UniqueName: \"kubernetes.io/projected/69a446fa-5d09-4f26-9b5d-72f3fbe42064-kube-api-access-8dchr\") pod \"dns-default-8z5vc\" (UID: \"69a446fa-5d09-4f26-9b5d-72f3fbe42064\") " pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:38.158935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.158881    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 18:21:38.159481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159441    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzp8k\" (UniqueName: \"kubernetes.io/projected/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-kube-api-access-jzp8k\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.159569 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159541    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprjg\" (UniqueName: \"kubernetes.io/projected/0a065905-b86e-4326-a7d8-ec336cb938c7-kube-api-access-cprjg\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.159653 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159630    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx5n\" (UniqueName: \"kubernetes.io/projected/445c9865-e1dc-467f-bb6f-f29e8c6b7f68-kube-api-access-sdx5n\") pod \"openshift-state-metrics-7bd7489957-2dktv\" (UID: \"445c9865-e1dc-467f-bb6f-f29e8c6b7f68\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2dktv"
Oct 31 18:21:38.159711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159643    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6bg2\" (UniqueName: \"kubernetes.io/projected/65c82ec5-b112-4e22-9e89-25b3a69fe6a2-kube-api-access-v6bg2\") pod \"kube-state-metrics-5dc8d7d89b-tdhb5\" (UID: \"65c82ec5-b112-4e22-9e89-25b3a69fe6a2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tdhb5"
Oct 31 18:21:38.159791 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159771    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64wql\" (UniqueName: \"kubernetes.io/projected/f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882-kube-api-access-64wql\") pod \"ingress-canary-zkb25\" (UID: \"f7f40051-3f1d-4d4f-9b06-b5f2cf7f5882\") " pod="openshift-ingress-canary/ingress-canary-zkb25"
Oct 31 18:21:38.159854 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159828    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 18:21:38.159909 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159890    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-ovn\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.159961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159945    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-ovn\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.159976    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-log-socket\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160018    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-log-socket\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160037    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmw9b\" (UniqueName: \"kubernetes.io/projected/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-kube-api-access-mmw9b\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.160157 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160062    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-systemd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160157 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160095    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-sys\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.160157 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160118    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-netns\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160157 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160145    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-system-cni-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.160350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160160    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5tp\" (UniqueName: \"kubernetes.io/projected/3497b74a-87c3-4f27-91b7-dac04680a875-kube-api-access-6m5tp\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:21:38.160350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160181    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-kubelet-config\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:38.160350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160210    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-registration-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.160350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160260    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-system-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160287    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-multus\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160308    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbbk\" (UniqueName: \"kubernetes.io/projected/3ad94d5c-d364-4d68-8795-70af86c61c4e-kube-api-access-8bbbk\") pod \"network-check-target-ztv9x\" (UID: \"3ad94d5c-d364-4d68-8795-70af86c61c4e\") " pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:38.160350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160318    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160346    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12b631c1-447d-43ee-9f5a-d77bfb04e03c-hosts-file\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160350    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmgd\" (UniqueName: \"kubernetes.io/projected/c8f7cf5d-858f-4ae2-b53e-c64881398c3f-kube-api-access-whmgd\") pod \"thanos-querier-695c5cddbb-m2rvq\" (UID: \"c8f7cf5d-858f-4ae2-b53e-c64881398c3f\") " pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160375    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-systemd\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160403    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-bin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160429    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-conf\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160437    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk88q\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160470    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-root\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160488    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/884384ea-01db-45f2-8c04-2163de1950ca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76f4cd98b8-hrnz4\" (UID: \"884384ea-01db-45f2-8c04-2163de1950ca\") " pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160505    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160510    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-root\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160543    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-systemd-units\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160565    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-sys\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160567    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160494    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7jq\" (UniqueName: \"kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq\") pod \"console-7986dc697c-f54gg\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") " pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160578    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrt52\" (UniqueName: \"kubernetes.io/projected/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kube-api-access-qrt52\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160608    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-netns\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160610    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-system-cni-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160571    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-wtmp\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160622    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-systemd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160634    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-systemd-units\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160657    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-node-exporter-wtmp\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160654    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-modprobe-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160688    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160692    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-multus\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160698    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-system-cni-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160693    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-registration-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160721    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160748    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-kubelet-config\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160753    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-systemd\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160794    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-cni-bin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160798    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-socket-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160795    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12b631c1-447d-43ee-9f5a-d77bfb04e03c-hosts-file\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160748    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-socket-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.160807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160825    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-modprobe-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160838    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-etc-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160872    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-kubelet\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160881    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160883    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzgw\" (UniqueName: \"kubernetes.io/projected/12b631c1-447d-43ee-9f5a-d77bfb04e03c-kube-api-access-mpzgw\") pod \"node-resolver-xjpst\" (UID: \"12b631c1-447d-43ee-9f5a-d77bfb04e03c\") " pod="openshift-dns/node-resolver-xjpst"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160723    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160846    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-run-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160901    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-k8s-cni-cncf-io\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160927    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-kubelet\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160927    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-etc-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160928    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrltf\" (UniqueName: \"kubernetes.io/projected/97e5d8ad-0765-4815-a495-53de9ca7b24b-kube-api-access-zrltf\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160910    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-conf\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160935    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-kubernetes\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160967    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-kubernetes\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160933    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-k8s-cni-cncf-io\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.160984    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-host\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161005    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-socket-dir-parent\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161031    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-cnibin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161034    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-host\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161067    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-run\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161079    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-cnibin\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161092    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161107    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-socket-dir-parent\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161119    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-dbus\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161124    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-run\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161148    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysconfig\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161154    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161184    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-lib-modules\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161154    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28594e99-2b75-44a6-8922-315e6b680700-dbus\") pod \"global-pull-secret-syncer-m56gf\" (UID: \"28594e99-2b75-44a6-8922-315e6b680700\") " pod="kube-system/global-pull-secret-syncer-m56gf"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161198    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysconfig\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161203    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-netns\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161226    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-hostroot\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161239    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-node-log\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161267    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-lib-modules\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161287    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-netns\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161300    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-node-log\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161308    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-device-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161327    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-bin\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161335    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-device-dir\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161300    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-hostroot\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161353    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161365    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-bin\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161372    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-slash\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161390    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-netd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161394    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-slash\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161410    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-cnibin\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161429    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3497b74a-87c3-4f27-91b7-dac04680a875-host-slash\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161440    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-etc-sysctl-d\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161443    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161444    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-cni-netd\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161479    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-kubelet\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161496    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3386381a-8b41-473b-9400-04d3d3eb2c7e-cnibin\") pod \"multus-additional-cni-plugins-rg4dh\" (UID: \"3386381a-8b41-473b-9400-04d3d3eb2c7e\") " pod="openshift-multus/multus-additional-cni-plugins-rg4dh"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161500    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-var-lib-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161511    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-var-lib-kubelet\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161521    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-var-lib-openvswitch\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161513    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a065905-b86e-4326-a7d8-ec336cb938c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzdz\" (UID: \"0a065905-b86e-4326-a7d8-ec336cb938c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161530    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e2f392-7651-4d4d-9c09-b4978aabcebf-host\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161548    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-conf-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161565    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-sys-fs\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161586    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-sys\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161590    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8e2f392-7651-4d4d-9c09-b4978aabcebf-host\") pod \"node-ca-rlwgx\" (UID: \"a8e2f392-7651-4d4d-9c09-b4978aabcebf\") " pod="openshift-image-registry/node-ca-rlwgx"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161612    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-etc-selinux\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161593    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-multus-conf-dir\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161621    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-sys-fs\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161499    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3497b74a-87c3-4f27-91b7-dac04680a875-host-slash\") pod \"iptables-alerter-25sgx\" (UID: \"3497b74a-87c3-4f27-91b7-dac04680a875\") " pod="openshift-network-operator/iptables-alerter-25sgx"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161881    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-multus-certs\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161887    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b37353a-7cbf-478e-b7c5-83a71f6c1986-etc-selinux\") pod \"aws-ebs-csi-driver-node-nq9mc\" (UID: \"8b37353a-7cbf-478e-b7c5-83a71f6c1986\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nq9mc"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.161965    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83fe51c6-ee5f-4ccf-a934-6d42a27d475b-sys\") pod \"node-exporter-fgkm2\" (UID: \"83fe51c6-ee5f-4ccf-a934-6d42a27d475b\") " pod="openshift-monitoring/node-exporter-fgkm2"
Oct 31 18:21:38.162017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.162064    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-host-run-multus-certs\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.164842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.162208    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-var-lib-kubelet\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.164842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.162242    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97e5d8ad-0765-4815-a495-53de9ca7b24b-crio-socket\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.164842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.162266    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-etc-kubernetes\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.164842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.162355    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe78dfe-d1f0-4196-9832-702532afcaf8-etc-kubernetes\") pod \"multus-bx85h\" (UID: \"1fe78dfe-d1f0-4196-9832-702532afcaf8\") " pod="openshift-multus/multus-bx85h"
Oct 31 18:21:38.164842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.162411    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be4d751-a0c8-4350-ac3f-8e12d3eca25c-var-lib-kubelet\") pod \"tuned-4g2hb\" (UID: \"8be4d751-a0c8-4350-ac3f-8e12d3eca25c\") " pod="openshift-cluster-node-tuning-operator/tuned-4g2hb"
Oct 31 18:21:38.164842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.162477    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97e5d8ad-0765-4815-a495-53de9ca7b24b-crio-socket\") pod \"insights-runtime-extractor-g8rpt\" (UID: \"97e5d8ad-0765-4815-a495-53de9ca7b24b\") " pod="openshift-insights/insights-runtime-extractor-g8rpt"
Oct 31 18:21:38.166182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.166160    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:21:38.258411 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:38.258392    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:21:39.163849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:39.161604    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8z5vc"
Oct 31 18:21:39.797496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:39.797440    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:40.004309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.003606    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq" event={"ID":"c8f7cf5d-858f-4ae2-b53e-c64881398c3f","Type":"ContainerStarted","Data":"a5824e562fb3abeb5ad2c97876087e9a617ac6c664fa37bc211d5380760c44ac"}
Oct 31 18:21:40.004309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.003678    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq" event={"ID":"c8f7cf5d-858f-4ae2-b53e-c64881398c3f","Type":"ContainerStarted","Data":"8edf05624748dcb2718465287f9b326fc4b9d51e1a99e1a3264bcedbeb1bc2ba"}
Oct 31 18:21:40.004309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.003703    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq" event={"ID":"c8f7cf5d-858f-4ae2-b53e-c64881398c3f","Type":"ContainerStarted","Data":"bb257b638fc05ce8585ded27ad3862c3d97f089debe12ea04b08ce4bf5ee0577"}
Oct 31 18:21:40.006424 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.006394    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ff57c786f-q298q" event={"ID":"cb3b9ece-b002-4973-95ed-862cd33ca046","Type":"ContainerStarted","Data":"b90623db602d1ee1a144fba5ecf58cea03343d3392bb46f4ea8e8f63ffec56cc"}
Oct 31 18:21:40.010868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.010846    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-695c5cddbb-m2rvq"
Oct 31 18:21:40.011097 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.011080    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerStarted","Data":"752fec395b89d68e474f859f68903272c88b26f0fec26e4af2db77580b67a39c"}
Oct 31 18:21:40.011182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.011124    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerStarted","Data":"f111eeed6cb3545a1cef8ef5032d15c8ee842d0a2b743fcff39311271b15ae3f"}
Oct 31 18:21:40.011182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.011151    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerStarted","Data":"65513beded7d0a35bfc2d89ca336ef9a940883b3e647b2ce9ca19ef964a1f009"}
Oct 31 18:21:40.011182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.011173    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerStarted","Data":"ec7b19f1f0a39f579b02c28b69f4b5f3db30ae22f8973041584a9ee0dfb7560c"}
Oct 31 18:21:40.011334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.011195    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerStarted","Data":"c407374833a1f99094d54061a70c4eec2a9d2ba8e9b298baf0edcf1da74f1b7a"}
Oct 31 18:21:40.575905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.575870    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8564c96dd4-rd45v"]
Oct 31 18:21:40.578321 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.578286    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.633544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.633519    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 18:21:40.637954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.637925    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8564c96dd4-rd45v"]
Oct 31 18:21:40.686553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.686515    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-serving-cert\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.686735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.686566    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-oauth-serving-cert\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.686735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.686665    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-tmp\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.686844 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.686814    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-config\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.686933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.686907    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xspkl\" (UniqueName: \"kubernetes.io/projected/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-kube-api-access-xspkl\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.687063 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.686948    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-oauth-config\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.687063 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.687015    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-trusted-ca-bundle\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.687159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.687069    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-service-ca\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.789494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.788803    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xspkl\" (UniqueName: \"kubernetes.io/projected/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-kube-api-access-xspkl\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.789494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.788855    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-oauth-config\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.789494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.788881    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-trusted-ca-bundle\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.789494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.788914    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-service-ca\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.789494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.788943    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-serving-cert\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.789494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.788970    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-oauth-serving-cert\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.789494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.789029    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-tmp\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.789494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.789076    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-config\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.790066 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.789875    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-config\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.790622 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.790585    5551 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 18:21:40.792383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.791729    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-oauth-serving-cert\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.792383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.792017    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-tmp\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.792383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.792158    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-trusted-ca-bundle\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.792743 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.792722    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-service-ca\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.795195 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.795171    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-oauth-config\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.797389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.797355    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-serving-cert\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.846956 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.846933    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xspkl\" (UniqueName: \"kubernetes.io/projected/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-kube-api-access-xspkl\") pod \"console-8564c96dd4-rd45v\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") " pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:40.892711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:40.892687    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:41.019963 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:41.019927    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerStarted","Data":"2ae661efa52aaf639b552e5fad037a8286c230a5a1b8b3a21d1dbd1929cd8a84"}
Oct 31 18:21:41.021932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:41.021906    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4" event={"ID":"884384ea-01db-45f2-8c04-2163de1950ca","Type":"ContainerStarted","Data":"8ade164d98bd6566983d499a155ae54f8d580377a454ec8677d0fb31e87be621"}
Oct 31 18:21:41.021993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:41.021969    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4" event={"ID":"884384ea-01db-45f2-8c04-2163de1950ca","Type":"ContainerStarted","Data":"0d264352e6b331b79585e30f25ccb782af51e07b872ef47f41958cbdad0edc5f"}
Oct 31 18:21:41.287433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:41.287358    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8564c96dd4-rd45v"]
Oct 31 18:21:41.292797 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:41.292760    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba6b5cc_9748_4af0_b42b_f44e1fc082fb.slice/crio-5a449f9f52e02717b6a5fcb309b4b83d1e115758400dcd0adfb57d11bda81a14 WatchSource:0}: Error finding container 5a449f9f52e02717b6a5fcb309b4b83d1e115758400dcd0adfb57d11bda81a14: Status 404 returned error can't find the container with id 5a449f9f52e02717b6a5fcb309b4b83d1e115758400dcd0adfb57d11bda81a14
Oct 31 18:21:41.469114 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:41.469086    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:42.028769 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.028729    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76f4cd98b8-hrnz4" event={"ID":"884384ea-01db-45f2-8c04-2163de1950ca","Type":"ContainerStarted","Data":"ee386cafcaa46eccb1a5620224fd16339d50893aa23360c9bfaf6c0d97083b3a"}
Oct 31 18:21:42.030441 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.030408    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564c96dd4-rd45v" event={"ID":"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb","Type":"ContainerStarted","Data":"bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9"}
Oct 31 18:21:42.030583 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.030480    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564c96dd4-rd45v" event={"ID":"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb","Type":"ContainerStarted","Data":"5a449f9f52e02717b6a5fcb309b4b83d1e115758400dcd0adfb57d11bda81a14"}
Oct 31 18:21:42.234266 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.234234    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:42.239945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.239917    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:42.331594 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.331517    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:21:42.379190 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.377838    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8564c96dd4-rd45v" podStartSLOduration=2.377820202 podStartE2EDuration="2.377820202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:42.25949396 +0000 UTC m=+5.490732251" watchObservedRunningTime="2026-10-31 18:21:42.377820202 +0000 UTC m=+5.609058493"
Oct 31 18:21:42.638180 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.638143    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:42.843845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.843812    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:42.844102 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:42.844083    5551 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 18:21:45.381166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.381124    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8564c96dd4-rd45v"]
Oct 31 18:21:45.431737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.431701    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cf4f64989-l7qqn"]
Oct 31 18:21:45.437402 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.437372    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.492170 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.492144    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf4f64989-l7qqn"]
Oct 31 18:21:45.539090 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.539060    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-trusted-ca-bundle\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.539233 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.539101    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/900fb99c-7003-4cf6-8d97-fdd71797d411-tmp\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.539233 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.539195    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-serving-cert\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.539339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.539264    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-service-ca\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.539339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.539329    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-oauth-config\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.539447 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.539356    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-oauth-serving-cert\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.539447 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.539377    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-console-config\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.539589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.539534    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc82s\" (UniqueName: \"kubernetes.io/projected/900fb99c-7003-4cf6-8d97-fdd71797d411-kube-api-access-wc82s\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.640987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.640919    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/900fb99c-7003-4cf6-8d97-fdd71797d411-tmp\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.640987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.640954    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-serving-cert\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.640987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.640984    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-service-ca\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.641194 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.641021    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-oauth-config\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.641194 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.641044    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-oauth-serving-cert\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.641194 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.641071    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-console-config\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.641348 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.641233    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc82s\" (UniqueName: \"kubernetes.io/projected/900fb99c-7003-4cf6-8d97-fdd71797d411-kube-api-access-wc82s\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.641403 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.641349    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-trusted-ca-bundle\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.641403 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.641385    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/900fb99c-7003-4cf6-8d97-fdd71797d411-tmp\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.641873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.641799    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-console-config\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.641873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.641869    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-service-ca\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.642093 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.642061    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-oauth-serving-cert\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.642579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.642557    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-trusted-ca-bundle\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.643805 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.643785    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-serving-cert\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.644048 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.644027    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-oauth-config\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.698369 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.698340    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc82s\" (UniqueName: \"kubernetes.io/projected/900fb99c-7003-4cf6-8d97-fdd71797d411-kube-api-access-wc82s\") pod \"console-5cf4f64989-l7qqn\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") " pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:45.752339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:45.752307    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:46.132308 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:46.132276    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf4f64989-l7qqn"]
Oct 31 18:21:46.136207 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:21:46.136177    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900fb99c_7003_4cf6_8d97_fdd71797d411.slice/crio-0c3ad1173af95df7e04d0a5c4a23949a74aca8d5f9e2968113cafe1deb709280 WatchSource:0}: Error finding container 0c3ad1173af95df7e04d0a5c4a23949a74aca8d5f9e2968113cafe1deb709280: Status 404 returned error can't find the container with id 0c3ad1173af95df7e04d0a5c4a23949a74aca8d5f9e2968113cafe1deb709280
Oct 31 18:21:46.608837 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:21:46.608754    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:21:46.790645 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:46.790614    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:21:47.054446 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:47.054402    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf4f64989-l7qqn" event={"ID":"900fb99c-7003-4cf6-8d97-fdd71797d411","Type":"ContainerStarted","Data":"d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0"}
Oct 31 18:21:47.054623 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:47.054486    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf4f64989-l7qqn" event={"ID":"900fb99c-7003-4cf6-8d97-fdd71797d411","Type":"ContainerStarted","Data":"0c3ad1173af95df7e04d0a5c4a23949a74aca8d5f9e2968113cafe1deb709280"}
Oct 31 18:21:47.160747 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:47.160708    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:21:47.169701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:47.169652    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5cf4f64989-l7qqn" podStartSLOduration=2.169613782 podStartE2EDuration="2.169613782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:47.168794297 +0000 UTC m=+10.400032587" watchObservedRunningTime="2026-10-31 18:21:47.169613782 +0000 UTC m=+10.400852072"
Oct 31 18:21:47.217113 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:47.217085    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:47.222838 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:47.222817    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:50.155967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:50.154666    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzdz"
Oct 31 18:21:52.643350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:52.643318    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:21:54.628821 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:54.628772    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:54.634682 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:54.634647    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:55.094573 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:55.094104    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:55.098210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:55.098183    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:21:55.265739 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:55.265030    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7986dc697c-f54gg"]
Oct 31 18:21:57.103472 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:57.103417    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-8cp96" event={"ID":"deca97c4-946e-4974-b30e-dd0595a94633","Type":"ContainerStarted","Data":"6b4b1449ccd8a96e1039f92a1920549c0bd5c27df434b3a6a08b6295ad953252"}
Oct 31 18:21:57.857643 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:21:57.857607    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:22:06.224910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:06.224882    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hgxrq"
Oct 31 18:22:06.793390 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:06.793331    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ztv9x"
Oct 31 18:22:07.103836 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:07.103763    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:22:07.122057 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:07.122036    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-8cp96"
Oct 31 18:22:10.403131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.403086    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8564c96dd4-rd45v" podUID="2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" containerName="console" containerID="cri-o://bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9" gracePeriod=15
Oct 31 18:22:10.681918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.681838    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:22:10.767928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.767888    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-oauth-config\") pod \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") "
Oct 31 18:22:10.768094 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.767955    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-config\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-config\") pod \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") "
Oct 31 18:22:10.768094 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.767973    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-trusted-ca-bundle\") pod \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") "
Oct 31 18:22:10.768094 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768080    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-tmp\") pod \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") "
Oct 31 18:22:10.768323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768119    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-service-ca\") pod \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") "
Oct 31 18:22:10.768323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768182    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-kube-api-access-xspkl\" (UniqueName: \"kubernetes.io/projected/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-kube-api-access-xspkl\") pod \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") "
Oct 31 18:22:10.768323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768267    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-oauth-serving-cert\") pod \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") "
Oct 31 18:22:10.768323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768312    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-serving-cert\") pod \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\" (UID: \"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb\") "
Oct 31 18:22:10.768575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768331    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-tmp" pod "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" (UID: "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:22:10.768575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768489    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-config" pod "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" (UID: "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:10.768681 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768611    5551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-config\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:10.768681 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768631    5551 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-tmp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:10.768681 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768620    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-service-ca" pod "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" (UID: "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:10.768808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768791    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-oauth-serving-cert" pod "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" (UID: "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:10.768866 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.768784    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-trusted-ca-bundle" pod "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" (UID: "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:10.770366 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.770342    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-oauth-config" pod "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" (UID: "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:10.770619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.770599    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-serving-cert" pod "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" (UID: "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:10.770696 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.770613    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-kube-api-access-xspkl" pod "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" (UID: "2ba6b5cc-9748-4af0-b42b-f44e1fc082fb"). InnerVolumeSpecName "kube-api-access-xspkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:22:10.869842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.869815    5551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-trusted-ca-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:10.869842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.869843    5551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:10.869993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.869852    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xspkl\" (UniqueName: \"kubernetes.io/projected/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-kube-api-access-xspkl\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:10.869993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.869862    5551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-oauth-serving-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:10.869993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.869871    5551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-serving-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:10.869993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:10.869880    5551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb-console-oauth-config\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:11.159673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.159639    5551 generic.go:356] "Generic (PLEG): container finished" podID="2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" containerID="bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9" exitCode=0
Oct 31 18:22:11.159863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.159691    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564c96dd4-rd45v"
Oct 31 18:22:11.159863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.159718    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564c96dd4-rd45v" event={"ID":"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb","Type":"ContainerDied","Data":"bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9"}
Oct 31 18:22:11.159863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.159770    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564c96dd4-rd45v" event={"ID":"2ba6b5cc-9748-4af0-b42b-f44e1fc082fb","Type":"ContainerDied","Data":"5a449f9f52e02717b6a5fcb309b4b83d1e115758400dcd0adfb57d11bda81a14"}
Oct 31 18:22:11.159863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.159810    5551 scope.go:122] "RemoveContainer" containerID="bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9"
Oct 31 18:22:11.167912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.167874    5551 scope.go:122] "RemoveContainer" containerID="bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9"
Oct 31 18:22:11.168175 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:22:11.168161    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9\": container with ID starting with bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9 not found: ID does not exist" containerID="bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9"
Oct 31 18:22:11.168217 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.168185    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9"} err="failed to get container status \"bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9\": rpc error: code = NotFound desc = could not find container \"bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9\": container with ID starting with bdd424b7dce84f1d26e9e6f37727639699f7f15d0ab9c1bca55e5dfa7910fdb9 not found: ID does not exist"
Oct 31 18:22:11.321881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.321859    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8564c96dd4-rd45v"]
Oct 31 18:22:11.371586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:11.371561    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8564c96dd4-rd45v"]
Oct 31 18:22:12.906861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:12.906828    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" path="/var/lib/kubelet/pods/2ba6b5cc-9748-4af0-b42b-f44e1fc082fb/volumes"
Oct 31 18:22:16.601929 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:22:16.601884    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:22:17.863604 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:17.863576    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:22:17.867251 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:17.867230    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7ff57c786f-q298q"
Oct 31 18:22:20.068761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.068725    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6964b87784-k5jl4"]
Oct 31 18:22:20.069192 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.069048    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" containerName="console"
Oct 31 18:22:20.069192 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.069059    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" containerName="console"
Oct 31 18:22:20.069192 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.069119    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba6b5cc-9748-4af0-b42b-f44e1fc082fb" containerName="console"
Oct 31 18:22:20.071417 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.071398    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.171619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.171585    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6964b87784-k5jl4"]
Oct 31 18:22:20.249253 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.249216    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ff2228e-c656-4077-bc59-93a40efd229f-tmp\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.249253 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.249260    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ff2228e-c656-4077-bc59-93a40efd229f-console-serving-cert\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.249514 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.249286    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-oauth-serving-cert\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.249514 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.249437    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6lk\" (UniqueName: \"kubernetes.io/projected/5ff2228e-c656-4077-bc59-93a40efd229f-kube-api-access-gg6lk\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.249514 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.249486    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-trusted-ca-bundle\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.249514 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.249509    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-service-ca\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.249651 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.249632    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ff2228e-c656-4077-bc59-93a40efd229f-console-oauth-config\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.249688 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.249671    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-console-config\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.287181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.287142    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7986dc697c-f54gg" podUID="519485a6-66ad-46f3-b5bc-76e5bc68315e" containerName="console" containerID="cri-o://e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3" gracePeriod=15
Oct 31 18:22:20.350138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350060    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-trusted-ca-bundle\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.350138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350110    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-service-ca\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.350323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350158    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ff2228e-c656-4077-bc59-93a40efd229f-console-oauth-config\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.350323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350181    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-console-config\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.350323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350207    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ff2228e-c656-4077-bc59-93a40efd229f-tmp\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.350323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350254    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ff2228e-c656-4077-bc59-93a40efd229f-console-serving-cert\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.350323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350298    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-oauth-serving-cert\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.350601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350355    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6lk\" (UniqueName: \"kubernetes.io/projected/5ff2228e-c656-4077-bc59-93a40efd229f-kube-api-access-gg6lk\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.350686 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350656    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ff2228e-c656-4077-bc59-93a40efd229f-tmp\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.351054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.350965    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-service-ca\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.351222 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.351112    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-console-config\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.351322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.351293    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-oauth-serving-cert\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.351410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.351358    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ff2228e-c656-4077-bc59-93a40efd229f-trusted-ca-bundle\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.352684 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.352664    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ff2228e-c656-4077-bc59-93a40efd229f-console-oauth-config\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.352809 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.352791    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ff2228e-c656-4077-bc59-93a40efd229f-console-serving-cert\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.407133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.407103    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6lk\" (UniqueName: \"kubernetes.io/projected/5ff2228e-c656-4077-bc59-93a40efd229f-kube-api-access-gg6lk\") pod \"console-6964b87784-k5jl4\" (UID: \"5ff2228e-c656-4077-bc59-93a40efd229f\") " pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.518523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.518496    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:22:20.652871 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.652838    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert\") pod \"519485a6-66ad-46f3-b5bc-76e5bc68315e\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") "
Oct 31 18:22:20.653043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.652878    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config\") pod \"519485a6-66ad-46f3-b5bc-76e5bc68315e\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") "
Oct 31 18:22:20.653043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.652902    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config\") pod \"519485a6-66ad-46f3-b5bc-76e5bc68315e\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") "
Oct 31 18:22:20.653043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.652953    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp\") pod \"519485a6-66ad-46f3-b5bc-76e5bc68315e\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") "
Oct 31 18:22:20.653043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.652978    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq\" (UniqueName: \"kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq\") pod \"519485a6-66ad-46f3-b5bc-76e5bc68315e\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") "
Oct 31 18:22:20.653043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.652997    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca\") pod \"519485a6-66ad-46f3-b5bc-76e5bc68315e\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") "
Oct 31 18:22:20.653300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.653043    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert\") pod \"519485a6-66ad-46f3-b5bc-76e5bc68315e\" (UID: \"519485a6-66ad-46f3-b5bc-76e5bc68315e\") "
Oct 31 18:22:20.653355 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.653321    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp" pod "519485a6-66ad-46f3-b5bc-76e5bc68315e" (UID: "519485a6-66ad-46f3-b5bc-76e5bc68315e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:22:20.653355 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.653340    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config" pod "519485a6-66ad-46f3-b5bc-76e5bc68315e" (UID: "519485a6-66ad-46f3-b5bc-76e5bc68315e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:20.653448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.653404    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca" pod "519485a6-66ad-46f3-b5bc-76e5bc68315e" (UID: "519485a6-66ad-46f3-b5bc-76e5bc68315e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:20.653589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.653564    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert" pod "519485a6-66ad-46f3-b5bc-76e5bc68315e" (UID: "519485a6-66ad-46f3-b5bc-76e5bc68315e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:20.655290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.655271    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config" pod "519485a6-66ad-46f3-b5bc-76e5bc68315e" (UID: "519485a6-66ad-46f3-b5bc-76e5bc68315e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:20.655393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.655370    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert" pod "519485a6-66ad-46f3-b5bc-76e5bc68315e" (UID: "519485a6-66ad-46f3-b5bc-76e5bc68315e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:20.655584 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.655566    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq" pod "519485a6-66ad-46f3-b5bc-76e5bc68315e" (UID: "519485a6-66ad-46f3-b5bc-76e5bc68315e"). InnerVolumeSpecName "kube-api-access-bp7jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:22:20.680787 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.680760    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:20.756948 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.756916    5551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-oauth-serving-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:20.756948 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.756948    5551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-serving-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:20.757166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.756965    5551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-oauth-config\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:20.757166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.756978    5551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-console-config\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:20.757166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.756992    5551 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/519485a6-66ad-46f3-b5bc-76e5bc68315e-tmp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:20.757166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.757007    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp7jq\" (UniqueName: \"kubernetes.io/projected/519485a6-66ad-46f3-b5bc-76e5bc68315e-kube-api-access-bp7jq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:20.757166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:20.757021    5551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/519485a6-66ad-46f3-b5bc-76e5bc68315e-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:21.070683 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.070609    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6964b87784-k5jl4"]
Oct 31 18:22:21.108078 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:22:21.108049    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff2228e_c656_4077_bc59_93a40efd229f.slice/crio-18a2910b4c6bd2619a1092bb398a2274a449d123cbc1ad7557ec4ff748bfe7d3 WatchSource:0}: Error finding container 18a2910b4c6bd2619a1092bb398a2274a449d123cbc1ad7557ec4ff748bfe7d3: Status 404 returned error can't find the container with id 18a2910b4c6bd2619a1092bb398a2274a449d123cbc1ad7557ec4ff748bfe7d3
Oct 31 18:22:21.200716 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.200626    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6964b87784-k5jl4" event={"ID":"5ff2228e-c656-4077-bc59-93a40efd229f","Type":"ContainerStarted","Data":"64389056ac49dbe19c51bcb1c4dd0fd98539d31b0f641b7c02781490a850326b"}
Oct 31 18:22:21.200716 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.200689    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6964b87784-k5jl4" event={"ID":"5ff2228e-c656-4077-bc59-93a40efd229f","Type":"ContainerStarted","Data":"18a2910b4c6bd2619a1092bb398a2274a449d123cbc1ad7557ec4ff748bfe7d3"}
Oct 31 18:22:21.202058 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.202035    5551 generic.go:356] "Generic (PLEG): container finished" podID="519485a6-66ad-46f3-b5bc-76e5bc68315e" containerID="e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3" exitCode=0
Oct 31 18:22:21.202181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.202074    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7986dc697c-f54gg"
Oct 31 18:22:21.202181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.202075    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7986dc697c-f54gg" event={"ID":"519485a6-66ad-46f3-b5bc-76e5bc68315e","Type":"ContainerDied","Data":"e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3"}
Oct 31 18:22:21.202252 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.202204    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7986dc697c-f54gg" event={"ID":"519485a6-66ad-46f3-b5bc-76e5bc68315e","Type":"ContainerDied","Data":"d24d184aa2ec9bcac439f2d949c9945598897bc7a9cfdaa271a427f0f703e74c"}
Oct 31 18:22:21.202252 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.202229    5551 scope.go:122] "RemoveContainer" containerID="e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3"
Oct 31 18:22:21.210914 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.210892    5551 scope.go:122] "RemoveContainer" containerID="e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3"
Oct 31 18:22:21.211180 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:22:21.211162    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3\": container with ID starting with e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3 not found: ID does not exist" containerID="e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3"
Oct 31 18:22:21.211225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.211189    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3"} err="failed to get container status \"e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3\": rpc error: code = NotFound desc = could not find container \"e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3\": container with ID starting with e05d2449255f376c92ed6cb241f4baed67523237ed2e343f28034f24888cfef3 not found: ID does not exist"
Oct 31 18:22:21.320699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.320652    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6964b87784-k5jl4" podStartSLOduration=1.320638736 podStartE2EDuration="1.320638736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:22:21.319866783 +0000 UTC m=+44.551105084" watchObservedRunningTime="2026-10-31 18:22:21.320638736 +0000 UTC m=+44.551877019"
Oct 31 18:22:21.428500 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.428453    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7986dc697c-f54gg"]
Oct 31 18:22:21.432376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:21.432351    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7986dc697c-f54gg"]
Oct 31 18:22:22.906439 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:22.906405    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519485a6-66ad-46f3-b5bc-76e5bc68315e" path="/var/lib/kubelet/pods/519485a6-66ad-46f3-b5bc-76e5bc68315e/volumes"
Oct 31 18:22:27.065739 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.065691    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:22:27.066204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.066059    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="alertmanager" containerID="cri-o://c407374833a1f99094d54061a70c4eec2a9d2ba8e9b298baf0edcf1da74f1b7a" gracePeriod=120
Oct 31 18:22:27.066204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.066066    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy-metric" containerID="cri-o://752fec395b89d68e474f859f68903272c88b26f0fec26e4af2db77580b67a39c" gracePeriod=120
Oct 31 18:22:27.066204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.066121    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy-web" containerID="cri-o://65513beded7d0a35bfc2d89ca336ef9a940883b3e647b2ce9ca19ef964a1f009" gracePeriod=120
Oct 31 18:22:27.066204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.066169    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="config-reloader" containerID="cri-o://ec7b19f1f0a39f579b02c28b69f4b5f3db30ae22f8973041584a9ee0dfb7560c" gracePeriod=120
Oct 31 18:22:27.066204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.066176    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="prom-label-proxy" containerID="cri-o://2ae661efa52aaf639b552e5fad037a8286c230a5a1b8b3a21d1dbd1929cd8a84" gracePeriod=120
Oct 31 18:22:27.066445 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.066203    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy" containerID="cri-o://f111eeed6cb3545a1cef8ef5032d15c8ee842d0a2b743fcff39311271b15ae3f" gracePeriod=120
Oct 31 18:22:27.227011 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.226987    5551 generic.go:356] "Generic (PLEG): container finished" podID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerID="2ae661efa52aaf639b552e5fad037a8286c230a5a1b8b3a21d1dbd1929cd8a84" exitCode=0
Oct 31 18:22:27.227011 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.227011    5551 generic.go:356] "Generic (PLEG): container finished" podID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerID="f111eeed6cb3545a1cef8ef5032d15c8ee842d0a2b743fcff39311271b15ae3f" exitCode=0
Oct 31 18:22:27.227137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.227020    5551 generic.go:356] "Generic (PLEG): container finished" podID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerID="ec7b19f1f0a39f579b02c28b69f4b5f3db30ae22f8973041584a9ee0dfb7560c" exitCode=0
Oct 31 18:22:27.227137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.227025    5551 generic.go:356] "Generic (PLEG): container finished" podID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerID="c407374833a1f99094d54061a70c4eec2a9d2ba8e9b298baf0edcf1da74f1b7a" exitCode=0
Oct 31 18:22:27.227137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.227061    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerDied","Data":"2ae661efa52aaf639b552e5fad037a8286c230a5a1b8b3a21d1dbd1929cd8a84"}
Oct 31 18:22:27.227226 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.227144    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerDied","Data":"f111eeed6cb3545a1cef8ef5032d15c8ee842d0a2b743fcff39311271b15ae3f"}
Oct 31 18:22:27.227226 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.227169    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerDied","Data":"ec7b19f1f0a39f579b02c28b69f4b5f3db30ae22f8973041584a9ee0dfb7560c"}
Oct 31 18:22:27.227226 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:27.227193    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerDied","Data":"c407374833a1f99094d54061a70c4eec2a9d2ba8e9b298baf0edcf1da74f1b7a"}
Oct 31 18:22:28.235798 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.235766    5551 generic.go:356] "Generic (PLEG): container finished" podID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerID="752fec395b89d68e474f859f68903272c88b26f0fec26e4af2db77580b67a39c" exitCode=0
Oct 31 18:22:28.235798 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.235795    5551 generic.go:356] "Generic (PLEG): container finished" podID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerID="65513beded7d0a35bfc2d89ca336ef9a940883b3e647b2ce9ca19ef964a1f009" exitCode=0
Oct 31 18:22:28.236221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.235829    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerDied","Data":"752fec395b89d68e474f859f68903272c88b26f0fec26e4af2db77580b67a39c"}
Oct 31 18:22:28.236221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.235907    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerDied","Data":"65513beded7d0a35bfc2d89ca336ef9a940883b3e647b2ce9ca19ef964a1f009"}
Oct 31 18:22:28.354956 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.354926    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:28.420148 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420084    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420148 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420140    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420171    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420211    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420244    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420280    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420319    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420350    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420382    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420418    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420486    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420513    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420548    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle\") pod \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\" (UID: \"2c130b20-c7b5-46a5-8c19-59ca36d172c8\") "
Oct 31 18:22:28.420704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420589    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:22:28.421017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.420792    5551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-main-db\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.423079 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.423042    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:28.423489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.423445    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:28.423715 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.423658    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:28.423930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.423907    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:22:28.424354 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.424309    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:28.424480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.424442    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "kube-api-access-bk88q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:22:28.424599 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.424559    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:28.424863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.424837    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:28.424951 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.424881    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:22:28.425408 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.425381    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:28.429056 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.429034    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:28.438879 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.435113    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config" pod "2c130b20-c7b5-46a5-8c19-59ca36d172c8" (UID: "2c130b20-c7b5-46a5-8c19-59ca36d172c8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:28.522099 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522075    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-metrics-client-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522099 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522095    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk88q\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-kube-api-access-bk88q\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522106    5551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522116    5551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522125    5551 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-out\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522135    5551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c130b20-c7b5-46a5-8c19-59ca36d172c8-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522144    5551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522173    5551 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-web-config\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522182    5551 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522191    5551 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c130b20-c7b5-46a5-8c19-59ca36d172c8-tls-assets\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522201    5551 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-cluster-tls-config\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:28.522232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:28.522209    5551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c130b20-c7b5-46a5-8c19-59ca36d172c8-secret-alertmanager-main-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:29.243831 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.243807    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.244269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.243801    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c130b20-c7b5-46a5-8c19-59ca36d172c8","Type":"ContainerDied","Data":"8b7fff8965badd296b83c5c42e37cd3cbc19e1263f5aeb185d9c54612146838a"}
Oct 31 18:22:29.244269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.243957    5551 scope.go:122] "RemoveContainer" containerID="2ae661efa52aaf639b552e5fad037a8286c230a5a1b8b3a21d1dbd1929cd8a84"
Oct 31 18:22:29.251907 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.251890    5551 scope.go:122] "RemoveContainer" containerID="752fec395b89d68e474f859f68903272c88b26f0fec26e4af2db77580b67a39c"
Oct 31 18:22:29.259030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.259009    5551 scope.go:122] "RemoveContainer" containerID="f111eeed6cb3545a1cef8ef5032d15c8ee842d0a2b743fcff39311271b15ae3f"
Oct 31 18:22:29.265803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.265780    5551 scope.go:122] "RemoveContainer" containerID="65513beded7d0a35bfc2d89ca336ef9a940883b3e647b2ce9ca19ef964a1f009"
Oct 31 18:22:29.272626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.272611    5551 scope.go:122] "RemoveContainer" containerID="ec7b19f1f0a39f579b02c28b69f4b5f3db30ae22f8973041584a9ee0dfb7560c"
Oct 31 18:22:29.287222 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.287188    5551 scope.go:122] "RemoveContainer" containerID="c407374833a1f99094d54061a70c4eec2a9d2ba8e9b298baf0edcf1da74f1b7a"
Oct 31 18:22:29.297683 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.297660    5551 scope.go:122] "RemoveContainer" containerID="23fb516d57f49fe77b6344c8f6d890a0ecdd2ac64f904ad7d452fedfcfb8c089"
Oct 31 18:22:29.458951 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.458873    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:22:29.508768 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.508732    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:22:29.509269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509247    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:22:29.509590 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509576    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy-web"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509590    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy-web"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509599    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy-metric"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509604    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy-metric"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509616    5551 cpu_manager.go:414] "containerMap: removing container" podUID="519485a6-66ad-46f3-b5bc-76e5bc68315e" containerName="console"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509622    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519485a6-66ad-46f3-b5bc-76e5bc68315e" containerName="console"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509631    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="alertmanager"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509636    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="alertmanager"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509641    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="init-config-reloader"
Oct 31 18:22:29.509646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509646    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="init-config-reloader"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509652    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509656    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509666    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="prom-label-proxy"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509671    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="prom-label-proxy"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509676    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="config-reloader"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509680    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="config-reloader"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509722    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy-web"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509729    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy-metric"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509736    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="519485a6-66ad-46f3-b5bc-76e5bc68315e" containerName="console"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509743    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="alertmanager"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509749    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="kube-rbac-proxy"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509754    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="prom-label-proxy"
Oct 31 18:22:29.509897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.509760    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" containerName="config-reloader"
Oct 31 18:22:29.533137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.533116    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.585723 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.585686    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 18:22:29.585723 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.585701    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 18:22:29.585723 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.585694    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 18:22:29.586026 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.585741    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l6bdm\""
Oct 31 18:22:29.586026 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.585753    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 18:22:29.586026 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.585757    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 18:22:29.586026 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.585786    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 18:22:29.586026 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.585794    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 18:22:29.586246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.586077    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 18:22:29.632248 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632211    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/169ebe1f-facb-4fbb-96cb-2535f0634218-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632271    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwq5\" (UniqueName: \"kubernetes.io/projected/169ebe1f-facb-4fbb-96cb-2535f0634218-kube-api-access-cjwq5\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632328    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/169ebe1f-facb-4fbb-96cb-2535f0634218-config-out\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632389    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/169ebe1f-facb-4fbb-96cb-2535f0634218-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632432    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-config-volume\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632450    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632524    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632553    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632573    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/169ebe1f-facb-4fbb-96cb-2535f0634218-tls-assets\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632801 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632594    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632801 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632688    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169ebe1f-facb-4fbb-96cb-2535f0634218-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632801 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632731    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.632898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.632807    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-web-config\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.635732 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.635673    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 18:22:29.733177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733117    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733146    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/169ebe1f-facb-4fbb-96cb-2535f0634218-tls-assets\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733168    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733187    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169ebe1f-facb-4fbb-96cb-2535f0634218-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733217    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733240    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-web-config\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733271    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/169ebe1f-facb-4fbb-96cb-2535f0634218-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733307    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwq5\" (UniqueName: \"kubernetes.io/projected/169ebe1f-facb-4fbb-96cb-2535f0634218-kube-api-access-cjwq5\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733339    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/169ebe1f-facb-4fbb-96cb-2535f0634218-config-out\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733362    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/169ebe1f-facb-4fbb-96cb-2535f0634218-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733396    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-config-volume\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733425    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.733954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.733482    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.734157 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.734094    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/169ebe1f-facb-4fbb-96cb-2535f0634218-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.734274 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.734215    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/169ebe1f-facb-4fbb-96cb-2535f0634218-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.735337 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.735282    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169ebe1f-facb-4fbb-96cb-2535f0634218-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.736295 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.736256    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/169ebe1f-facb-4fbb-96cb-2535f0634218-tls-assets\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.736494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.736472    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-web-config\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.736700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.736653    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.736700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.736678    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.736921 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.736901    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.737002 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.736917    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/169ebe1f-facb-4fbb-96cb-2535f0634218-config-out\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.737501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.737483    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.737851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.737835    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-config-volume\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.738789 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.738719    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/169ebe1f-facb-4fbb-96cb-2535f0634218-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.740848 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.740825    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:22:29.789996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.789965    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwq5\" (UniqueName: \"kubernetes.io/projected/169ebe1f-facb-4fbb-96cb-2535f0634218-kube-api-access-cjwq5\") pod \"alertmanager-main-0\" (UID: \"169ebe1f-facb-4fbb-96cb-2535f0634218\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:29.955121 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:29.955076    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 18:22:30.463438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:30.463410    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 18:22:30.517229 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:22:30.517198    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169ebe1f_facb_4fbb_96cb_2535f0634218.slice/crio-678b4bb907d8fc45ee3bdd5078fd297ba1abea18e992f98d3866774da5bbf14c WatchSource:0}: Error finding container 678b4bb907d8fc45ee3bdd5078fd297ba1abea18e992f98d3866774da5bbf14c: Status 404 returned error can't find the container with id 678b4bb907d8fc45ee3bdd5078fd297ba1abea18e992f98d3866774da5bbf14c
Oct 31 18:22:30.681594 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:30.681550    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:30.681742 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:30.681623    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:30.686214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:30.686195    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:30.906963 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:30.906928    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c130b20-c7b5-46a5-8c19-59ca36d172c8" path="/var/lib/kubelet/pods/2c130b20-c7b5-46a5-8c19-59ca36d172c8/volumes"
Oct 31 18:22:31.254275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:31.254243    5551 generic.go:356] "Generic (PLEG): container finished" podID="169ebe1f-facb-4fbb-96cb-2535f0634218" containerID="08638805d0e445aea1fce1a260d7a49b9c129a3e45392e66717aa2febe186153" exitCode=0
Oct 31 18:22:31.254425 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:31.254333    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"169ebe1f-facb-4fbb-96cb-2535f0634218","Type":"ContainerDied","Data":"08638805d0e445aea1fce1a260d7a49b9c129a3e45392e66717aa2febe186153"}
Oct 31 18:22:31.254425 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:31.254397    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"169ebe1f-facb-4fbb-96cb-2535f0634218","Type":"ContainerStarted","Data":"678b4bb907d8fc45ee3bdd5078fd297ba1abea18e992f98d3866774da5bbf14c"}
Oct 31 18:22:31.258166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:31.258140    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6964b87784-k5jl4"
Oct 31 18:22:31.514091 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:31.514067    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf4f64989-l7qqn"]
Oct 31 18:22:32.265536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:32.265494    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"169ebe1f-facb-4fbb-96cb-2535f0634218","Type":"ContainerStarted","Data":"601bcd4f6e05002d9f3922cbc3892168ddf2120f055d6d4cf3781dabf183311b"}
Oct 31 18:22:32.265721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:32.265605    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"169ebe1f-facb-4fbb-96cb-2535f0634218","Type":"ContainerStarted","Data":"c7ee23eb77e0740ad20248a6a05569c6196e04a9b79f756ed9772309ca7c1c82"}
Oct 31 18:22:32.265721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:32.265633    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"169ebe1f-facb-4fbb-96cb-2535f0634218","Type":"ContainerStarted","Data":"231beb928cf6022340c9b19598d03999d5b8de8874ba0b5e03dddc3b9b855d76"}
Oct 31 18:22:32.265721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:32.265662    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"169ebe1f-facb-4fbb-96cb-2535f0634218","Type":"ContainerStarted","Data":"6009b516c89e9187f6ecb1e773d5956fbb4638eb454cccfd35b840cc5b77a560"}
Oct 31 18:22:32.265721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:32.265687    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"169ebe1f-facb-4fbb-96cb-2535f0634218","Type":"ContainerStarted","Data":"0c6f8ca49a9e499d2551354317d1e544275c98c22594a6cb908a7a605bc24402"}
Oct 31 18:22:32.265721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:32.265710    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"169ebe1f-facb-4fbb-96cb-2535f0634218","Type":"ContainerStarted","Data":"843918d283ec6524e2cf248df5248465c54bc3a759cdc9fc14c2a05c0c2e4b9d"}
Oct 31 18:22:32.387211 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:32.387158    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.37506606 podStartE2EDuration="3.387144953s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.078893ms" isStatefulPod=false podCreationTimestamp="2026-10-31 18:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:22:32.385725372 +0000 UTC m=+55.616963697" watchObservedRunningTime="2026-10-31 18:22:32.387144953 +0000 UTC m=+55.618383234"
Oct 31 18:22:46.603177 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:22:46.603144    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:22:56.533527 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.533452    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5cf4f64989-l7qqn" podUID="900fb99c-7003-4cf6-8d97-fdd71797d411" containerName="console" containerID="cri-o://d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0" gracePeriod=15
Oct 31 18:22:56.781142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.781120    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:22:56.872860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.872801    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-service-ca\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-service-ca\") pod \"900fb99c-7003-4cf6-8d97-fdd71797d411\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") "
Oct 31 18:22:56.872860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.872834    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900fb99c-7003-4cf6-8d97-fdd71797d411-tmp\" (UniqueName: \"kubernetes.io/empty-dir/900fb99c-7003-4cf6-8d97-fdd71797d411-tmp\") pod \"900fb99c-7003-4cf6-8d97-fdd71797d411\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") "
Oct 31 18:22:56.873045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.872871    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-console-config\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-console-config\") pod \"900fb99c-7003-4cf6-8d97-fdd71797d411\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") "
Oct 31 18:22:56.873045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.872897    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-serving-cert\") pod \"900fb99c-7003-4cf6-8d97-fdd71797d411\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") "
Oct 31 18:22:56.873045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.872923    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-oauth-config\") pod \"900fb99c-7003-4cf6-8d97-fdd71797d411\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") "
Oct 31 18:22:56.873045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.872984    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900fb99c-7003-4cf6-8d97-fdd71797d411-kube-api-access-wc82s\" (UniqueName: \"kubernetes.io/projected/900fb99c-7003-4cf6-8d97-fdd71797d411-kube-api-access-wc82s\") pod \"900fb99c-7003-4cf6-8d97-fdd71797d411\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") "
Oct 31 18:22:56.873045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.873018    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-trusted-ca-bundle\") pod \"900fb99c-7003-4cf6-8d97-fdd71797d411\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") "
Oct 31 18:22:56.873290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.873046    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-oauth-serving-cert\") pod \"900fb99c-7003-4cf6-8d97-fdd71797d411\" (UID: \"900fb99c-7003-4cf6-8d97-fdd71797d411\") "
Oct 31 18:22:56.873290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.873139    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900fb99c-7003-4cf6-8d97-fdd71797d411-tmp" pod "900fb99c-7003-4cf6-8d97-fdd71797d411" (UID: "900fb99c-7003-4cf6-8d97-fdd71797d411"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:22:56.873290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.873258    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-service-ca" pod "900fb99c-7003-4cf6-8d97-fdd71797d411" (UID: "900fb99c-7003-4cf6-8d97-fdd71797d411"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:56.873414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.873307    5551 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/900fb99c-7003-4cf6-8d97-fdd71797d411-tmp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:56.873414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.873379    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-console-config" pod "900fb99c-7003-4cf6-8d97-fdd71797d411" (UID: "900fb99c-7003-4cf6-8d97-fdd71797d411"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:56.873602 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.873452    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-oauth-serving-cert" pod "900fb99c-7003-4cf6-8d97-fdd71797d411" (UID: "900fb99c-7003-4cf6-8d97-fdd71797d411"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:56.873602 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.873489    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-trusted-ca-bundle" pod "900fb99c-7003-4cf6-8d97-fdd71797d411" (UID: "900fb99c-7003-4cf6-8d97-fdd71797d411"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:56.875141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.875111    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-serving-cert" pod "900fb99c-7003-4cf6-8d97-fdd71797d411" (UID: "900fb99c-7003-4cf6-8d97-fdd71797d411"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:56.875237 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.875211    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900fb99c-7003-4cf6-8d97-fdd71797d411-kube-api-access-wc82s" pod "900fb99c-7003-4cf6-8d97-fdd71797d411" (UID: "900fb99c-7003-4cf6-8d97-fdd71797d411"). InnerVolumeSpecName "kube-api-access-wc82s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:22:56.875331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.875312    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-oauth-config" pod "900fb99c-7003-4cf6-8d97-fdd71797d411" (UID: "900fb99c-7003-4cf6-8d97-fdd71797d411"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:56.974671 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.974642    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc82s\" (UniqueName: \"kubernetes.io/projected/900fb99c-7003-4cf6-8d97-fdd71797d411-kube-api-access-wc82s\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:56.974671 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.974669    5551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-trusted-ca-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:56.974863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.974679    5551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-oauth-serving-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:56.974863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.974689    5551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:56.974863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.974697    5551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/900fb99c-7003-4cf6-8d97-fdd71797d411-console-config\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:56.974863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.974705    5551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-serving-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:56.974863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:56.974713    5551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900fb99c-7003-4cf6-8d97-fdd71797d411-console-oauth-config\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:57.356846 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.356811    5551 generic.go:356] "Generic (PLEG): container finished" podID="900fb99c-7003-4cf6-8d97-fdd71797d411" containerID="d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0" exitCode=0
Oct 31 18:22:57.357029 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.356855    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf4f64989-l7qqn"
Oct 31 18:22:57.357029 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.356896    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf4f64989-l7qqn" event={"ID":"900fb99c-7003-4cf6-8d97-fdd71797d411","Type":"ContainerDied","Data":"d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0"}
Oct 31 18:22:57.357029 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.356951    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf4f64989-l7qqn" event={"ID":"900fb99c-7003-4cf6-8d97-fdd71797d411","Type":"ContainerDied","Data":"0c3ad1173af95df7e04d0a5c4a23949a74aca8d5f9e2968113cafe1deb709280"}
Oct 31 18:22:57.357029 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.356971    5551 scope.go:122] "RemoveContainer" containerID="d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0"
Oct 31 18:22:57.365044 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.365027    5551 scope.go:122] "RemoveContainer" containerID="d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0"
Oct 31 18:22:57.365310 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:22:57.365287    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0\": container with ID starting with d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0 not found: ID does not exist" containerID="d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0"
Oct 31 18:22:57.365356 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.365320    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0"} err="failed to get container status \"d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0\": rpc error: code = NotFound desc = could not find container \"d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0\": container with ID starting with d35818d43ef0c88436038dba20570a5d4ddc15e8e18d33879dc764258fa01fa0 not found: ID does not exist"
Oct 31 18:22:57.519296 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.519273    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf4f64989-l7qqn"]
Oct 31 18:22:57.568784 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:57.568766    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cf4f64989-l7qqn"]
Oct 31 18:22:58.906233 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:22:58.906200    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900fb99c-7003-4cf6-8d97-fdd71797d411" path="/var/lib/kubelet/pods/900fb99c-7003-4cf6-8d97-fdd71797d411/volumes"
Oct 31 18:23:16.603491 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:23:16.603398    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:23:46.603450 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:23:46.603417    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:24:16.604903 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:24:16.604866    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:24:37.295706 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.295674    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"]
Oct 31 18:24:37.296196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.296020    5551 cpu_manager.go:414] "containerMap: removing container" podUID="900fb99c-7003-4cf6-8d97-fdd71797d411" containerName="console"
Oct 31 18:24:37.296196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.296031    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900fb99c-7003-4cf6-8d97-fdd71797d411" containerName="console"
Oct 31 18:24:37.296196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.296101    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="900fb99c-7003-4cf6-8d97-fdd71797d411" containerName="console"
Oct 31 18:24:37.298231 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.298213    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"]
Oct 31 18:24:37.298368 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.298351    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.300330 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.300309    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.347034 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.347009    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2t72\" (UniqueName: \"kubernetes.io/projected/8205297a-7593-4712-b393-e1fabfff8a15-kube-api-access-f2t72\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.347130 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.347041    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.347130 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.347064    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.347130 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.347116    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.347247 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.347232    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.347281 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.347269    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tx8\" (UniqueName: \"kubernetes.io/projected/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-kube-api-access-l2tx8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.351226 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.351201    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-qsgtq\""
Oct 31 18:24:37.351358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.351346    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 18:24:37.351405 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.351372    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 18:24:37.400839 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.400816    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-vwmf4\""
Oct 31 18:24:37.409837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.409813    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"]
Oct 31 18:24:37.414032 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.413910    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"]
Oct 31 18:24:37.447913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.447880    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tx8\" (UniqueName: \"kubernetes.io/projected/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-kube-api-access-l2tx8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.448087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.447923    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2t72\" (UniqueName: \"kubernetes.io/projected/8205297a-7593-4712-b393-e1fabfff8a15-kube-api-access-f2t72\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.448087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.447944    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.448087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.447968    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.448087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.447994    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.448087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.448072    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.448440 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.448396    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.448440 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.448425    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.448584 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.448485    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.448584 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.448502    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.513068 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.513042    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2t72\" (UniqueName: \"kubernetes.io/projected/8205297a-7593-4712-b393-e1fabfff8a15-kube-api-access-f2t72\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.515569 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.515544    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tx8\" (UniqueName: \"kubernetes.io/projected/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-kube-api-access-l2tx8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:37.611909 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.611835    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:24:37.616731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:37.616716    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:24:38.033717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:38.033682    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"]
Oct 31 18:24:38.035747 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:38.035635    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"]
Oct 31 18:24:38.039293 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:38.039239    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8205297a_7593_4712_b393_e1fabfff8a15.slice/crio-8de3512c50551afe15c9b7c16e19595608d239b1c82bd6ae736e75d7cb3885d2 WatchSource:0}: Error finding container 8de3512c50551afe15c9b7c16e19595608d239b1c82bd6ae736e75d7cb3885d2: Status 404 returned error can't find the container with id 8de3512c50551afe15c9b7c16e19595608d239b1c82bd6ae736e75d7cb3885d2
Oct 31 18:24:38.039929 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:38.039906    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c67b89_e2b8_4b5f_a44d_170d24943bbd.slice/crio-30260bf11185972bab22c6088e4888ae34a85d912c3cc6ab5c5200b018e1580f WatchSource:0}: Error finding container 30260bf11185972bab22c6088e4888ae34a85d912c3cc6ab5c5200b018e1580f: Status 404 returned error can't find the container with id 30260bf11185972bab22c6088e4888ae34a85d912c3cc6ab5c5200b018e1580f
Oct 31 18:24:38.721613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:38.721576    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj" event={"ID":"c3c67b89-e2b8-4b5f-a44d-170d24943bbd","Type":"ContainerStarted","Data":"30260bf11185972bab22c6088e4888ae34a85d912c3cc6ab5c5200b018e1580f"}
Oct 31 18:24:38.722720 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:38.722684    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq" event={"ID":"8205297a-7593-4712-b393-e1fabfff8a15","Type":"ContainerStarted","Data":"8de3512c50551afe15c9b7c16e19595608d239b1c82bd6ae736e75d7cb3885d2"}
Oct 31 18:24:39.878627 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:39.878592    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"]
Oct 31 18:24:39.881713 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:39.881691    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:39.968406 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:39.968367    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:39.968626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:39.968423    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgxbw\" (UniqueName: \"kubernetes.io/projected/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-kube-api-access-hgxbw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:39.968626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:39.968479    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:39.987363 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:39.987331    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"]
Oct 31 18:24:39.988089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:39.988071    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-vsd8h\""
Oct 31 18:24:40.070138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.069121    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:40.070138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.069172    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgxbw\" (UniqueName: \"kubernetes.io/projected/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-kube-api-access-hgxbw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:40.070138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.069210    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:40.070138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.069830    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:40.070138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.069850    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:40.127626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.127591    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgxbw\" (UniqueName: \"kubernetes.io/projected/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-kube-api-access-hgxbw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:40.199327 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.199247    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:24:40.583818 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.583738    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"]
Oct 31 18:24:40.589857 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:40.589818    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c25851_ff28_4e0d_b1a9_1c7b976252fc.slice/crio-0e7214ff232ee641ba0ba78164012287fbf97f27f59173edd5a2f0f9879a964d WatchSource:0}: Error finding container 0e7214ff232ee641ba0ba78164012287fbf97f27f59173edd5a2f0f9879a964d: Status 404 returned error can't find the container with id 0e7214ff232ee641ba0ba78164012287fbf97f27f59173edd5a2f0f9879a964d
Oct 31 18:24:40.733470 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:40.733411    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4" event={"ID":"b6c25851-ff28-4e0d-b1a9-1c7b976252fc","Type":"ContainerStarted","Data":"0e7214ff232ee641ba0ba78164012287fbf97f27f59173edd5a2f0f9879a964d"}
Oct 31 18:24:41.154522 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.154486    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"]
Oct 31 18:24:41.157675 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.157652    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.179816 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.179781    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.179938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.179848    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.179999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.179980    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmcm\" (UniqueName: \"kubernetes.io/projected/66e1f424-62df-4cb4-981d-d356ae245cea-kube-api-access-lxmcm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.209770 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.209727    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-wj7nz\""
Oct 31 18:24:41.218239 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.218211    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"]
Oct 31 18:24:41.258438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.258411    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"]
Oct 31 18:24:41.261376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.261359    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.280808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.280783    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.280942 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.280849    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmcm\" (UniqueName: \"kubernetes.io/projected/66e1f424-62df-4cb4-981d-d356ae245cea-kube-api-access-lxmcm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.280942 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.280874    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flslq\" (UniqueName: \"kubernetes.io/projected/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-kube-api-access-flslq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.280942 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.280920    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.281085 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.280953    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.281143 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.281117    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.281214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.281194    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.281427 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.281408    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.313353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.313325    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-5msf8\""
Oct 31 18:24:41.320947 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.320922    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"]
Oct 31 18:24:41.338988 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.338966    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmcm\" (UniqueName: \"kubernetes.io/projected/66e1f424-62df-4cb4-981d-d356ae245cea-kube-api-access-lxmcm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.364200 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.364170    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"]
Oct 31 18:24:41.367347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.367321    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.382566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.382542    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4mp\" (UniqueName: \"kubernetes.io/projected/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-kube-api-access-5g4mp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.382669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.382595    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.382730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.382713    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.382793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.382753    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flslq\" (UniqueName: \"kubernetes.io/projected/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-kube-api-access-flslq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.382863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.382834    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.382925 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.382887    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.383211 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.383179    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.383310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.383228    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.421728 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.421647    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-g4flc\""
Oct 31 18:24:41.425055 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.425023    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"]
Oct 31 18:24:41.439998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.439967    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flslq\" (UniqueName: \"kubernetes.io/projected/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-kube-api-access-flslq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.457479 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.457423    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"]
Oct 31 18:24:41.459979 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.459956    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.474515 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.474494    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:24:41.483825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.483793    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42fc\" (UniqueName: \"kubernetes.io/projected/be02ed07-a67b-46ca-bfa5-a2f627901b08-kube-api-access-v42fc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.483922 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.483869    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4mp\" (UniqueName: \"kubernetes.io/projected/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-kube-api-access-5g4mp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.483922 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.483902    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.484019 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.483931    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.484019 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.483997    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.484115 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.484024    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.484390 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.484366    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.484982 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.484960    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.520334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.520302    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-nz9d4\""
Oct 31 18:24:41.520492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.520353    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"]
Oct 31 18:24:41.545437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.545118    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4mp\" (UniqueName: \"kubernetes.io/projected/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-kube-api-access-5g4mp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.575352 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.575313    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:24:41.585284 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.585252    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.585417 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.585339    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42fc\" (UniqueName: \"kubernetes.io/projected/be02ed07-a67b-46ca-bfa5-a2f627901b08-kube-api-access-v42fc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.585417 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.585405    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.586299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.586278    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.586600 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.586579    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.645930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.645623    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42fc\" (UniqueName: \"kubernetes.io/projected/be02ed07-a67b-46ca-bfa5-a2f627901b08-kube-api-access-v42fc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.683900 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.683600    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:24:41.779529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.778722    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:24:41.872184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.872151    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"]
Oct 31 18:24:41.877076 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:41.877050    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e1f424_62df_4cb4_981d_d356ae245cea.slice/crio-889c56e670b92be77f1f3c09da5e0bb7a4ad1058ea8858aeecbbad1f101fb405 WatchSource:0}: Error finding container 889c56e670b92be77f1f3c09da5e0bb7a4ad1058ea8858aeecbbad1f101fb405: Status 404 returned error can't find the container with id 889c56e670b92be77f1f3c09da5e0bb7a4ad1058ea8858aeecbbad1f101fb405
Oct 31 18:24:41.971885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:41.971760    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"]
Oct 31 18:24:41.975678 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:41.975624    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe13bc5b_c59c_42b1_adc1_c4eb76883d20.slice/crio-b32bf9e065b7d9aa023f94caa458f03d9a0efad86e98c6cfa54b6abae3d2dfa3 WatchSource:0}: Error finding container b32bf9e065b7d9aa023f94caa458f03d9a0efad86e98c6cfa54b6abae3d2dfa3: Status 404 returned error can't find the container with id b32bf9e065b7d9aa023f94caa458f03d9a0efad86e98c6cfa54b6abae3d2dfa3
Oct 31 18:24:42.081474 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:42.076622    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"]
Oct 31 18:24:42.085281 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:42.085254    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f42e19_9ab6_4302_b4c5_fef0abe42c35.slice/crio-889efbc90756f73f3800612397fd9cf34485f20569f7a8a1d97edf94d9b16f07 WatchSource:0}: Error finding container 889efbc90756f73f3800612397fd9cf34485f20569f7a8a1d97edf94d9b16f07: Status 404 returned error can't find the container with id 889efbc90756f73f3800612397fd9cf34485f20569f7a8a1d97edf94d9b16f07
Oct 31 18:24:42.174499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:42.173209    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"]
Oct 31 18:24:42.176866 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:42.176835    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe02ed07_a67b_46ca_bfa5_a2f627901b08.slice/crio-841619f5760c90f62a06a72a167e869e3b1035d1dab3378899bca1edaeee3631 WatchSource:0}: Error finding container 841619f5760c90f62a06a72a167e869e3b1035d1dab3378899bca1edaeee3631: Status 404 returned error can't find the container with id 841619f5760c90f62a06a72a167e869e3b1035d1dab3378899bca1edaeee3631
Oct 31 18:24:42.747717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:42.747601    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh" event={"ID":"be02ed07-a67b-46ca-bfa5-a2f627901b08","Type":"ContainerStarted","Data":"841619f5760c90f62a06a72a167e869e3b1035d1dab3378899bca1edaeee3631"}
Oct 31 18:24:42.749596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:42.749546    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz" event={"ID":"66e1f424-62df-4cb4-981d-d356ae245cea","Type":"ContainerStarted","Data":"889c56e670b92be77f1f3c09da5e0bb7a4ad1058ea8858aeecbbad1f101fb405"}
Oct 31 18:24:42.750977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:42.750951    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf" event={"ID":"fe13bc5b-c59c-42b1-adc1-c4eb76883d20","Type":"ContainerStarted","Data":"b32bf9e065b7d9aa023f94caa458f03d9a0efad86e98c6cfa54b6abae3d2dfa3"}
Oct 31 18:24:42.752216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:42.752192    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99" event={"ID":"c8f42e19-9ab6-4302-b4c5-fef0abe42c35","Type":"ContainerStarted","Data":"889efbc90756f73f3800612397fd9cf34485f20569f7a8a1d97edf94d9b16f07"}
Oct 31 18:24:43.179153 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.179117    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"]
Oct 31 18:24:43.202258 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.202227    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.241565 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.241521    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"]
Oct 31 18:24:43.304049 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.304015    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.304220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.304107    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.304220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.304166    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xvj\" (UniqueName: \"kubernetes.io/projected/e08f54bd-3381-4b65-95ac-4536eb9cc011-kube-api-access-b5xvj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.401715 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.399262    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"]
Oct 31 18:24:43.406498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.405352    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.406498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.405438    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.406498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.405521    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xvj\" (UniqueName: \"kubernetes.io/projected/e08f54bd-3381-4b65-95ac-4536eb9cc011-kube-api-access-b5xvj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.406498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.406002    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.406498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.406331    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.407416 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.407051    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.460788 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.460711    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"]
Oct 31 18:24:43.468935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.468853    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xvj\" (UniqueName: \"kubernetes.io/projected/e08f54bd-3381-4b65-95ac-4536eb9cc011-kube-api-access-b5xvj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.506491 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.506335    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.506491 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.506398    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.506967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.506867    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkmf\" (UniqueName: \"kubernetes.io/projected/cc19cf95-db91-4751-ad9d-a3c2118f903a-kube-api-access-nzkmf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.520956 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.520721    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:24:43.608581 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.608038    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkmf\" (UniqueName: \"kubernetes.io/projected/cc19cf95-db91-4751-ad9d-a3c2118f903a-kube-api-access-nzkmf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.608581 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.608097    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.608581 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.608146    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.608958 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.608587    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.609220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.609167    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.666074 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.666041    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkmf\" (UniqueName: \"kubernetes.io/projected/cc19cf95-db91-4751-ad9d-a3c2118f903a-kube-api-access-nzkmf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.725404 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.725120    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:24:43.929205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:43.929072    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"]
Oct 31 18:24:43.937553 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:43.937504    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08f54bd_3381_4b65_95ac_4536eb9cc011.slice/crio-fdc62e5c77ad773cf7f96a96597831f1b8a5a0ae8064884f9ba4c2c88314dd96 WatchSource:0}: Error finding container fdc62e5c77ad773cf7f96a96597831f1b8a5a0ae8064884f9ba4c2c88314dd96: Status 404 returned error can't find the container with id fdc62e5c77ad773cf7f96a96597831f1b8a5a0ae8064884f9ba4c2c88314dd96
Oct 31 18:24:44.127143 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.127069    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"]
Oct 31 18:24:44.146615 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:44.146575    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc19cf95_db91_4751_ad9d_a3c2118f903a.slice/crio-c44573e86291027b4c1a1c856a8b838f8213a7c91c8a760a113b084312da5e0b WatchSource:0}: Error finding container c44573e86291027b4c1a1c856a8b838f8213a7c91c8a760a113b084312da5e0b: Status 404 returned error can't find the container with id c44573e86291027b4c1a1c856a8b838f8213a7c91c8a760a113b084312da5e0b
Oct 31 18:24:44.620685 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.619244    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"]
Oct 31 18:24:44.626592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.624961    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.681129 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.681100    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nsw28\""
Oct 31 18:24:44.681301 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.681245    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 18:24:44.681372 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.681328    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 18:24:44.681436 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.681416    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728079    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728126    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728188    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728257    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728282    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728326    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728352    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728380    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.728642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.728416    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf67v\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-kube-api-access-qf67v\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.733261 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.733203    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"]
Oct 31 18:24:44.768166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.768063    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8" event={"ID":"e08f54bd-3381-4b65-95ac-4536eb9cc011","Type":"ContainerStarted","Data":"fdc62e5c77ad773cf7f96a96597831f1b8a5a0ae8064884f9ba4c2c88314dd96"}
Oct 31 18:24:44.770084 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.770056    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm" event={"ID":"cc19cf95-db91-4751-ad9d-a3c2118f903a","Type":"ContainerStarted","Data":"c44573e86291027b4c1a1c856a8b838f8213a7c91c8a760a113b084312da5e0b"}
Oct 31 18:24:44.830380 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830348    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.830578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830390    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.830578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830432    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.830578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830478    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.830578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830507    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.830578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830540    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf67v\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-kube-api-access-qf67v\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.830824 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830595    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.830824 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830622    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.830824 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.830693    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.831527 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.831500    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.831774 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.831751    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.832286 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.832263    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.832760 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.832740    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.833721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.833701    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.835816 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.835760    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.857833 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.853687    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.891396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.891193    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf67v\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-kube-api-access-qf67v\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.892000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.891945    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rbrxs\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.924902 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.924865    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"]
Oct 31 18:24:44.927957 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.927934    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:44.945239 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.945214    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:44.981496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.980412    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-kc79c\""
Oct 31 18:24:44.987312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:44.987268    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"]
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.032781    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.032817    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.032837    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.032876    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.032923    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.032959    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.032983    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.033008    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khshq\" (UniqueName: \"kubernetes.io/projected/3c19da20-58a9-4a4d-9fe4-7ea13270d595-kube-api-access-khshq\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.034803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.033058    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134438    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134518    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134575    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134607    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khshq\" (UniqueName: \"kubernetes.io/projected/3c19da20-58a9-4a4d-9fe4-7ea13270d595-kube-api-access-khshq\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134661    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134723    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134755    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134782    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.134845    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.135682    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.135961    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.136670    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.137711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.137420    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.138932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.138826    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.141087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.140840    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.154863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.154802    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.205244 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.205185    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c19da20-58a9-4a4d-9fe4-7ea13270d595-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.208360 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.208331    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khshq\" (UniqueName: \"kubernetes.io/projected/3c19da20-58a9-4a4d-9fe4-7ea13270d595-kube-api-access-khshq\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls\" (UID: \"3c19da20-58a9-4a4d-9fe4-7ea13270d595\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.244437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.244405    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:45.358918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.358822    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"]
Oct 31 18:24:45.674438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.674388    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"]
Oct 31 18:24:45.684013 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:24:45.683938    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c19da20_58a9_4a4d_9fe4_7ea13270d595.slice/crio-d54fbc0c24da0642ea81db176d46807bcf2bb9541e384907d9bf93b14e56bc2b WatchSource:0}: Error finding container d54fbc0c24da0642ea81db176d46807bcf2bb9541e384907d9bf93b14e56bc2b: Status 404 returned error can't find the container with id d54fbc0c24da0642ea81db176d46807bcf2bb9541e384907d9bf93b14e56bc2b
Oct 31 18:24:45.782133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.782094    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" event={"ID":"9525d3e3-170d-42d1-a7cf-89a87b80d0fc","Type":"ContainerStarted","Data":"69a0b7a0bb5eb059a9982f9e13a0a1e13e5a4594c53aa0f4648d7a30bb664424"}
Oct 31 18:24:45.785203 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:45.785171    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls" event={"ID":"3c19da20-58a9-4a4d-9fe4-7ea13270d595","Type":"ContainerStarted","Data":"d54fbc0c24da0642ea81db176d46807bcf2bb9541e384907d9bf93b14e56bc2b"}
Oct 31 18:24:46.633083 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:24:46.633025    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:24:54.881817 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:54.881776    5551 generic.go:356] "Generic (PLEG): container finished" podID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerID="2efbd23bb2b34143e6bc0653d247860b2b1697b65b74846e7d668113d1afd713" exitCode=0
Oct 31 18:24:54.882321 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:54.881913    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj" event={"ID":"c3c67b89-e2b8-4b5f-a44d-170d24943bbd","Type":"ContainerDied","Data":"2efbd23bb2b34143e6bc0653d247860b2b1697b65b74846e7d668113d1afd713"}
Oct 31 18:24:54.888273 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:54.888247    5551 generic.go:356] "Generic (PLEG): container finished" podID="8205297a-7593-4712-b393-e1fabfff8a15" containerID="2f6d066119ecaed49dcc250aef4625e92896ec523d895b68452b0f2fdfb96881" exitCode=0
Oct 31 18:24:54.888365 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:54.888353    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq" event={"ID":"8205297a-7593-4712-b393-e1fabfff8a15","Type":"ContainerDied","Data":"2f6d066119ecaed49dcc250aef4625e92896ec523d895b68452b0f2fdfb96881"}
Oct 31 18:24:54.894997 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:54.894880    5551 generic.go:356] "Generic (PLEG): container finished" podID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerID="add9652316b91a66589946fdd7a8d2d63ac72ba12c1ef22066df827b18ea559a" exitCode=0
Oct 31 18:24:54.894997 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:54.894960    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf" event={"ID":"fe13bc5b-c59c-42b1-adc1-c4eb76883d20","Type":"ContainerDied","Data":"add9652316b91a66589946fdd7a8d2d63ac72ba12c1ef22066df827b18ea559a"}
Oct 31 18:24:54.899202 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:54.899172    5551 generic.go:356] "Generic (PLEG): container finished" podID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerID="49198958f81da24906fc2ccb38f37cab066cc52e54d6715651fb7720da85be0b" exitCode=0
Oct 31 18:24:54.899265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:54.899219    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99" event={"ID":"c8f42e19-9ab6-4302-b4c5-fef0abe42c35","Type":"ContainerDied","Data":"49198958f81da24906fc2ccb38f37cab066cc52e54d6715651fb7720da85be0b"}
Oct 31 18:24:55.161002 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.160963    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:24:55.161124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.161054    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:24:55.161124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.161097    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:24:55.579911 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.579882    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:24:55.579995 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.579952    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:24:55.580036 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.579999    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:24:55.916535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.915771    5551 generic.go:356] "Generic (PLEG): container finished" podID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerID="8f6bad21514e1217e2cdd8b98e418353dcba02ee6a208870c4e14db205201aad" exitCode=0
Oct 31 18:24:55.916535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.915908    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm" event={"ID":"cc19cf95-db91-4751-ad9d-a3c2118f903a","Type":"ContainerDied","Data":"8f6bad21514e1217e2cdd8b98e418353dcba02ee6a208870c4e14db205201aad"}
Oct 31 18:24:55.924252 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.924211    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls" event={"ID":"3c19da20-58a9-4a4d-9fe4-7ea13270d595","Type":"ContainerStarted","Data":"fefbc070a6d90e80ded8678a6390d9543ef7ed51d79a3ef8e2180362487d9c37"}
Oct 31 18:24:55.926051 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.926025    5551 generic.go:356] "Generic (PLEG): container finished" podID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerID="1ad7f56f9e5505ebaa8dab0fc764b0546b17d12efd45fdf0f39d92ac8f27a8d5" exitCode=0
Oct 31 18:24:55.926158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.926099    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh" event={"ID":"be02ed07-a67b-46ca-bfa5-a2f627901b08","Type":"ContainerDied","Data":"1ad7f56f9e5505ebaa8dab0fc764b0546b17d12efd45fdf0f39d92ac8f27a8d5"}
Oct 31 18:24:55.928669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.928597    5551 generic.go:356] "Generic (PLEG): container finished" podID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerID="fec908115d18d80692c3c51fc0dc9183d2d33a59cd24bad4945016c2c24499e8" exitCode=0
Oct 31 18:24:55.928750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.928708    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8" event={"ID":"e08f54bd-3381-4b65-95ac-4536eb9cc011","Type":"ContainerDied","Data":"fec908115d18d80692c3c51fc0dc9183d2d33a59cd24bad4945016c2c24499e8"}
Oct 31 18:24:55.931725 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.931162    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" event={"ID":"9525d3e3-170d-42d1-a7cf-89a87b80d0fc","Type":"ContainerStarted","Data":"ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780"}
Oct 31 18:24:55.945407 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:55.945386    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:56.245719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.245635    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:56.251784 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.251756    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:56.373016 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.372968    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" podStartSLOduration=2.590106114 podStartE2EDuration="12.372952315s" totalImagesPullingTime="9.782846201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:24:56.371190882 +0000 UTC m=+199.602429182" watchObservedRunningTime="2026-10-31 18:24:56.372952315 +0000 UTC m=+199.604190604"
Oct 31 18:24:56.488897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.488410    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls" podStartSLOduration=2.599105326 podStartE2EDuration="12.488390746s" totalImagesPullingTime="9.88928542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:24:56.486408408 +0000 UTC m=+199.717646711" watchObservedRunningTime="2026-10-31 18:24:56.488390746 +0000 UTC m=+199.719629030"
Oct 31 18:24:56.938919 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.938891    5551 generic.go:356] "Generic (PLEG): container finished" podID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerID="81809b22f3743732581ebacd51d9926ae7dbfb0105642f9546f81fc7777f81e1" exitCode=0
Oct 31 18:24:56.939304 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.939003    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf" event={"ID":"fe13bc5b-c59c-42b1-adc1-c4eb76883d20","Type":"ContainerDied","Data":"81809b22f3743732581ebacd51d9926ae7dbfb0105642f9546f81fc7777f81e1"}
Oct 31 18:24:56.941063 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.941037    5551 generic.go:356] "Generic (PLEG): container finished" podID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerID="9e2fbdffc3ce305f571e19a6762c768f0e355c86262b291dd5c84ca435a7ece1" exitCode=0
Oct 31 18:24:56.941925 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.941899    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4" event={"ID":"b6c25851-ff28-4e0d-b1a9-1c7b976252fc","Type":"ContainerDied","Data":"9e2fbdffc3ce305f571e19a6762c768f0e355c86262b291dd5c84ca435a7ece1"}
Oct 31 18:24:56.942026 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.941953    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:56.942996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.942949    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls"
Oct 31 18:24:56.952399 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:56.952376    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:57.950212 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:57.950167    5551 generic.go:356] "Generic (PLEG): container finished" podID="66e1f424-62df-4cb4-981d-d356ae245cea" containerID="27437f0c80304007f8f224c8b42a48aafbf019be485bfa777e004bb0297e78d4" exitCode=0
Oct 31 18:24:57.950691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:57.950317    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz" event={"ID":"66e1f424-62df-4cb4-981d-d356ae245cea","Type":"ContainerDied","Data":"27437f0c80304007f8f224c8b42a48aafbf019be485bfa777e004bb0297e78d4"}
Oct 31 18:24:57.955055 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:57.954570    5551 generic.go:356] "Generic (PLEG): container finished" podID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerID="cb5c9344bf3d9f21df74013f45ccf8bf55872465b4f328d38a2523e498a0f1c3" exitCode=0
Oct 31 18:24:57.955055 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:57.954976    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99" event={"ID":"c8f42e19-9ab6-4302-b4c5-fef0abe42c35","Type":"ContainerDied","Data":"cb5c9344bf3d9f21df74013f45ccf8bf55872465b4f328d38a2523e498a0f1c3"}
Oct 31 18:24:57.955227 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:57.955180    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:57.957512 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:57.957412    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:24:58.963998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:58.963068    5551 generic.go:356] "Generic (PLEG): container finished" podID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerID="0323346a353f735ce6e847bdb883cc3e5386aa0e59cecf95ebedba2b4bbe3a60" exitCode=0
Oct 31 18:24:58.963998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:58.963180    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4" event={"ID":"b6c25851-ff28-4e0d-b1a9-1c7b976252fc","Type":"ContainerDied","Data":"0323346a353f735ce6e847bdb883cc3e5386aa0e59cecf95ebedba2b4bbe3a60"}
Oct 31 18:24:58.981806 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:58.981770    5551 generic.go:356] "Generic (PLEG): container finished" podID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerID="3789fa494731fb58f81f8392389401e92c473833919f1ffab229582ecc0f8620" exitCode=0
Oct 31 18:24:58.983529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:58.983502    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh" event={"ID":"be02ed07-a67b-46ca-bfa5-a2f627901b08","Type":"ContainerDied","Data":"3789fa494731fb58f81f8392389401e92c473833919f1ffab229582ecc0f8620"}
Oct 31 18:24:59.993819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:24:59.993366    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj" event={"ID":"c3c67b89-e2b8-4b5f-a44d-170d24943bbd","Type":"ContainerStarted","Data":"0dd6466a4cb97862c8d258a3245f6af1b8c5f7c63e0ae8b1ade126a845bf1de3"}
Oct 31 18:25:01.002026 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:01.001993    5551 generic.go:356] "Generic (PLEG): container finished" podID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerID="0dd6466a4cb97862c8d258a3245f6af1b8c5f7c63e0ae8b1ade126a845bf1de3" exitCode=0
Oct 31 18:25:01.002420 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:01.002122    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj" event={"ID":"c3c67b89-e2b8-4b5f-a44d-170d24943bbd","Type":"ContainerDied","Data":"0dd6466a4cb97862c8d258a3245f6af1b8c5f7c63e0ae8b1ade126a845bf1de3"}
Oct 31 18:25:01.006376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:01.005717    5551 generic.go:356] "Generic (PLEG): container finished" podID="8205297a-7593-4712-b393-e1fabfff8a15" containerID="401e82802943fccf86f089a9e37c28d90c0037e3a68452430df8e6b00595ab1d" exitCode=0
Oct 31 18:25:01.006376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:01.005773    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq" event={"ID":"8205297a-7593-4712-b393-e1fabfff8a15","Type":"ContainerDied","Data":"401e82802943fccf86f089a9e37c28d90c0037e3a68452430df8e6b00595ab1d"}
Oct 31 18:25:02.014877 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:02.014792    5551 generic.go:356] "Generic (PLEG): container finished" podID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerID="6d3d3864336cb07da4cc52ca9854fefd4f5dc23ba5a9e95c715179fe30454c95" exitCode=0
Oct 31 18:25:02.015335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:02.014904    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm" event={"ID":"cc19cf95-db91-4751-ad9d-a3c2118f903a","Type":"ContainerDied","Data":"6d3d3864336cb07da4cc52ca9854fefd4f5dc23ba5a9e95c715179fe30454c95"}
Oct 31 18:25:02.017414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:02.017383    5551 generic.go:356] "Generic (PLEG): container finished" podID="66e1f424-62df-4cb4-981d-d356ae245cea" containerID="68af7fc254733fd56ed20325e39113753cb6361b53997a8d74859bce8b74ae1f" exitCode=0
Oct 31 18:25:02.017558 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:02.017487    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz" event={"ID":"66e1f424-62df-4cb4-981d-d356ae245cea","Type":"ContainerDied","Data":"68af7fc254733fd56ed20325e39113753cb6361b53997a8d74859bce8b74ae1f"}
Oct 31 18:25:02.020280 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:02.020255    5551 generic.go:356] "Generic (PLEG): container finished" podID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerID="a98e6111b6963b994f311e7fac9ee8b97d7ed737420a33444e75af98e1231c1d" exitCode=0
Oct 31 18:25:02.020371 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:02.020289    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8" event={"ID":"e08f54bd-3381-4b65-95ac-4536eb9cc011","Type":"ContainerDied","Data":"a98e6111b6963b994f311e7fac9ee8b97d7ed737420a33444e75af98e1231c1d"}
Oct 31 18:25:15.165863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:15.165825    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf" event={"ID":"fe13bc5b-c59c-42b1-adc1-c4eb76883d20","Type":"ContainerStarted","Data":"16f34478e70933581de3cb7f35978f130b9f94c29fe45f2330c4c285e9b54632"}
Oct 31 18:25:15.285381 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:15.284891    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf" podStartSLOduration=2.824019682 podStartE2EDuration="34.284869495s" totalImagesPullingTime="31.460849813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:24:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:15.283135216 +0000 UTC m=+218.514373505" watchObservedRunningTime="2026-10-31 18:25:15.284869495 +0000 UTC m=+218.516107789"
Oct 31 18:25:16.182802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:16.182249    5551 generic.go:356] "Generic (PLEG): container finished" podID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerID="16f34478e70933581de3cb7f35978f130b9f94c29fe45f2330c4c285e9b54632" exitCode=0
Oct 31 18:25:16.182802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:16.182356    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf" event={"ID":"fe13bc5b-c59c-42b1-adc1-c4eb76883d20","Type":"ContainerDied","Data":"16f34478e70933581de3cb7f35978f130b9f94c29fe45f2330c4c285e9b54632"}
Oct 31 18:25:16.195156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:16.195119    5551 generic.go:356] "Generic (PLEG): container finished" podID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerID="b23c7998a5fc311fd8a53856e4fb5dc5a85a584886663dbf31c5d7e71bdf35c0" exitCode=0
Oct 31 18:25:16.195293 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:16.195226    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99" event={"ID":"c8f42e19-9ab6-4302-b4c5-fef0abe42c35","Type":"ContainerDied","Data":"b23c7998a5fc311fd8a53856e4fb5dc5a85a584886663dbf31c5d7e71bdf35c0"}
Oct 31 18:25:16.619495 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:25:16.618845    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:25:17.207869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.207825    5551 generic.go:356] "Generic (PLEG): container finished" podID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerID="1908b2c3a615c28dbed83495b43a6141fcb938bf312ec49da7f7ac021bc706e6" exitCode=0
Oct 31 18:25:17.209125 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.209078    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4" event={"ID":"b6c25851-ff28-4e0d-b1a9-1c7b976252fc","Type":"ContainerDied","Data":"1908b2c3a615c28dbed83495b43a6141fcb938bf312ec49da7f7ac021bc706e6"}
Oct 31 18:25:17.451826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.451542    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:25:17.463684 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.463619    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:25:17.511878 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.511818    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-bundle\") pod \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") "
Oct 31 18:25:17.512015 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.511909    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-util\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-util\") pod \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") "
Oct 31 18:25:17.512015 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.511951    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-util\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-util\") pod \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") "
Oct 31 18:25:17.512015 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.511990    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-bundle\") pod \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") "
Oct 31 18:25:17.512188 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.512047    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-kube-api-access-5g4mp\" (UniqueName: \"kubernetes.io/projected/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-kube-api-access-5g4mp\") pod \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\" (UID: \"c8f42e19-9ab6-4302-b4c5-fef0abe42c35\") "
Oct 31 18:25:17.512188 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.512080    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-kube-api-access-flslq\" (UniqueName: \"kubernetes.io/projected/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-kube-api-access-flslq\") pod \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\" (UID: \"fe13bc5b-c59c-42b1-adc1-c4eb76883d20\") "
Oct 31 18:25:17.513265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.513223    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-bundle" pod "fe13bc5b-c59c-42b1-adc1-c4eb76883d20" (UID: "fe13bc5b-c59c-42b1-adc1-c4eb76883d20"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:17.514667 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.513445    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-bundle" pod "c8f42e19-9ab6-4302-b4c5-fef0abe42c35" (UID: "c8f42e19-9ab6-4302-b4c5-fef0abe42c35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:17.515908 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.515196    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-kube-api-access-flslq" pod "fe13bc5b-c59c-42b1-adc1-c4eb76883d20" (UID: "fe13bc5b-c59c-42b1-adc1-c4eb76883d20"). InnerVolumeSpecName "kube-api-access-flslq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:17.516068 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.516013    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-kube-api-access-5g4mp" pod "c8f42e19-9ab6-4302-b4c5-fef0abe42c35" (UID: "c8f42e19-9ab6-4302-b4c5-fef0abe42c35"). InnerVolumeSpecName "kube-api-access-5g4mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:17.519309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.519246    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-util" pod "c8f42e19-9ab6-4302-b4c5-fef0abe42c35" (UID: "c8f42e19-9ab6-4302-b4c5-fef0abe42c35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:17.520179 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.520154    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-util" pod "fe13bc5b-c59c-42b1-adc1-c4eb76883d20" (UID: "fe13bc5b-c59c-42b1-adc1-c4eb76883d20"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:17.613337 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.613288    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:17.613337 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.613340    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:17.613636 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.613355    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:17.613636 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.613369    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:17.613636 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.613385    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g4mp\" (UniqueName: \"kubernetes.io/projected/c8f42e19-9ab6-4302-b4c5-fef0abe42c35-kube-api-access-5g4mp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:17.613636 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:17.613403    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flslq\" (UniqueName: \"kubernetes.io/projected/fe13bc5b-c59c-42b1-adc1-c4eb76883d20-kube-api-access-flslq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:18.224416 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.224375    5551 generic.go:356] "Generic (PLEG): container finished" podID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerID="5e50721aa7455aa33fe0f47e6751004294106b89fcdedcaa5df465d3728c6907" exitCode=0
Oct 31 18:25:18.224780 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.224528    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh" event={"ID":"be02ed07-a67b-46ca-bfa5-a2f627901b08","Type":"ContainerDied","Data":"5e50721aa7455aa33fe0f47e6751004294106b89fcdedcaa5df465d3728c6907"}
Oct 31 18:25:18.228796 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.228769    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf" event={"ID":"fe13bc5b-c59c-42b1-adc1-c4eb76883d20","Type":"ContainerDied","Data":"b32bf9e065b7d9aa023f94caa458f03d9a0efad86e98c6cfa54b6abae3d2dfa3"}
Oct 31 18:25:18.228902 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.228818    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b32bf9e065b7d9aa023f94caa458f03d9a0efad86e98c6cfa54b6abae3d2dfa3"
Oct 31 18:25:18.228902 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.228885    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hlqqf"
Oct 31 18:25:18.237471 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.237431    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99"
Oct 31 18:25:18.237583 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.237544    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbzh99" event={"ID":"c8f42e19-9ab6-4302-b4c5-fef0abe42c35","Type":"ContainerDied","Data":"889efbc90756f73f3800612397fd9cf34485f20569f7a8a1d97edf94d9b16f07"}
Oct 31 18:25:18.237641 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.237580    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="889efbc90756f73f3800612397fd9cf34485f20569f7a8a1d97edf94d9b16f07"
Oct 31 18:25:18.433138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.433035    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:25:18.523069 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.523027    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-util\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-util\") pod \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") "
Oct 31 18:25:18.523069 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.523072    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-kube-api-access-hgxbw\" (UniqueName: \"kubernetes.io/projected/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-kube-api-access-hgxbw\") pod \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") "
Oct 31 18:25:18.523322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.523192    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-bundle\") pod \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\" (UID: \"b6c25851-ff28-4e0d-b1a9-1c7b976252fc\") "
Oct 31 18:25:18.525980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.525948    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-bundle" pod "b6c25851-ff28-4e0d-b1a9-1c7b976252fc" (UID: "b6c25851-ff28-4e0d-b1a9-1c7b976252fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:18.527918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.527889    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-kube-api-access-hgxbw" pod "b6c25851-ff28-4e0d-b1a9-1c7b976252fc" (UID: "b6c25851-ff28-4e0d-b1a9-1c7b976252fc"). InnerVolumeSpecName "kube-api-access-hgxbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:18.530613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.530579    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-util" pod "b6c25851-ff28-4e0d-b1a9-1c7b976252fc" (UID: "b6c25851-ff28-4e0d-b1a9-1c7b976252fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:18.624225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.624190    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:18.624225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.624226    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:18.624447 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:18.624243    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgxbw\" (UniqueName: \"kubernetes.io/projected/b6c25851-ff28-4e0d-b1a9-1c7b976252fc-kube-api-access-hgxbw\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:19.247968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.247937    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4"
Oct 31 18:25:19.248574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.248361    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xmkt4" event={"ID":"b6c25851-ff28-4e0d-b1a9-1c7b976252fc","Type":"ContainerDied","Data":"0e7214ff232ee641ba0ba78164012287fbf97f27f59173edd5a2f0f9879a964d"}
Oct 31 18:25:19.248574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.248396    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e7214ff232ee641ba0ba78164012287fbf97f27f59173edd5a2f0f9879a964d"
Oct 31 18:25:19.434822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.434749    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:25:19.533437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.533401    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-util\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-util\") pod \"be02ed07-a67b-46ca-bfa5-a2f627901b08\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") "
Oct 31 18:25:19.533625 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.533493    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-bundle\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-bundle\") pod \"be02ed07-a67b-46ca-bfa5-a2f627901b08\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") "
Oct 31 18:25:19.533625 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.533620    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be02ed07-a67b-46ca-bfa5-a2f627901b08-kube-api-access-v42fc\" (UniqueName: \"kubernetes.io/projected/be02ed07-a67b-46ca-bfa5-a2f627901b08-kube-api-access-v42fc\") pod \"be02ed07-a67b-46ca-bfa5-a2f627901b08\" (UID: \"be02ed07-a67b-46ca-bfa5-a2f627901b08\") "
Oct 31 18:25:19.541521 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.541491    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-bundle" pod "be02ed07-a67b-46ca-bfa5-a2f627901b08" (UID: "be02ed07-a67b-46ca-bfa5-a2f627901b08"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:19.541644 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.541597    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-util" pod "be02ed07-a67b-46ca-bfa5-a2f627901b08" (UID: "be02ed07-a67b-46ca-bfa5-a2f627901b08"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:19.544412 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.544383    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be02ed07-a67b-46ca-bfa5-a2f627901b08-kube-api-access-v42fc" pod "be02ed07-a67b-46ca-bfa5-a2f627901b08" (UID: "be02ed07-a67b-46ca-bfa5-a2f627901b08"). InnerVolumeSpecName "kube-api-access-v42fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:19.634930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.634890    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:19.634930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.634929    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be02ed07-a67b-46ca-bfa5-a2f627901b08-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:19.635187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:19.634945    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42fc\" (UniqueName: \"kubernetes.io/projected/be02ed07-a67b-46ca-bfa5-a2f627901b08-kube-api-access-v42fc\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:20.254106 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.254079    5551 generic.go:356] "Generic (PLEG): container finished" podID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerID="bae4e8f28537d6d579fe96e6ffb0926974fd5de54281aad35b7987ca2183a5ef" exitCode=0
Oct 31 18:25:20.254571 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.254137    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8" event={"ID":"e08f54bd-3381-4b65-95ac-4536eb9cc011","Type":"ContainerDied","Data":"bae4e8f28537d6d579fe96e6ffb0926974fd5de54281aad35b7987ca2183a5ef"}
Oct 31 18:25:20.255790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.255774    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:25:20.256514 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.256494    5551 generic.go:356] "Generic (PLEG): container finished" podID="8205297a-7593-4712-b393-e1fabfff8a15" containerID="30b6780418bfc9fd5b7b59e134b344ca4fb9c34be3b937d2d32c50e11f33e8ee" exitCode=1
Oct 31 18:25:20.256614 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.256583    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq" event={"ID":"8205297a-7593-4712-b393-e1fabfff8a15","Type":"ContainerDied","Data":"30b6780418bfc9fd5b7b59e134b344ca4fb9c34be3b937d2d32c50e11f33e8ee"}
Oct 31 18:25:20.258786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.258766    5551 generic.go:356] "Generic (PLEG): container finished" podID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerID="409c79c8e3333132cea76d374a819d4bd11ccf924e0bcf43fcfe83806db148d1" exitCode=0
Oct 31 18:25:20.258875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.258852    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj" event={"ID":"c3c67b89-e2b8-4b5f-a44d-170d24943bbd","Type":"ContainerDied","Data":"409c79c8e3333132cea76d374a819d4bd11ccf924e0bcf43fcfe83806db148d1"}
Oct 31 18:25:20.261036 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.261016    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm" event={"ID":"cc19cf95-db91-4751-ad9d-a3c2118f903a","Type":"ContainerStarted","Data":"4044217c580d52e6e3ae9ddb94d4fc641beaaf612b55dd78284201b53e912e99"}
Oct 31 18:25:20.262999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.262975    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh" event={"ID":"be02ed07-a67b-46ca-bfa5-a2f627901b08","Type":"ContainerDied","Data":"841619f5760c90f62a06a72a167e869e3b1035d1dab3378899bca1edaeee3631"}
Oct 31 18:25:20.262999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.262991    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amkkfh"
Oct 31 18:25:20.263142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.263013    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="841619f5760c90f62a06a72a167e869e3b1035d1dab3378899bca1edaeee3631"
Oct 31 18:25:20.265120 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.265097    5551 generic.go:356] "Generic (PLEG): container finished" podID="66e1f424-62df-4cb4-981d-d356ae245cea" containerID="63a63ee5b17cb35c45547e0a6e86500894ce3e8c2da832ce1b7f6e9369ab39e6" exitCode=0
Oct 31 18:25:20.265211 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.265129    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz" event={"ID":"66e1f424-62df-4cb4-981d-d356ae245cea","Type":"ContainerDied","Data":"63a63ee5b17cb35c45547e0a6e86500894ce3e8c2da832ce1b7f6e9369ab39e6"}
Oct 31 18:25:20.595752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:20.595703    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm" podStartSLOduration=3.3761064689999998 podStartE2EDuration="37.595689583s" totalImagesPullingTime="34.219583114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:24:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:20.594677625 +0000 UTC m=+223.825915913" watchObservedRunningTime="2026-10-31 18:25:20.595689583 +0000 UTC m=+223.826927864"
Oct 31 18:25:21.272559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.272388    5551 generic.go:356] "Generic (PLEG): container finished" podID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerID="4044217c580d52e6e3ae9ddb94d4fc641beaaf612b55dd78284201b53e912e99" exitCode=0
Oct 31 18:25:21.272559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.272508    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm" event={"ID":"cc19cf95-db91-4751-ad9d-a3c2118f903a","Type":"ContainerDied","Data":"4044217c580d52e6e3ae9ddb94d4fc641beaaf612b55dd78284201b53e912e99"}
Oct 31 18:25:21.432855 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.432821    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:25:21.463032 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.462999    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-bundle\") pod \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") "
Oct 31 18:25:21.463183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.463076    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-kube-api-access-l2tx8\" (UniqueName: \"kubernetes.io/projected/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-kube-api-access-l2tx8\") pod \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") "
Oct 31 18:25:21.463183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.463143    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-util\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-util\") pod \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\" (UID: \"c3c67b89-e2b8-4b5f-a44d-170d24943bbd\") "
Oct 31 18:25:21.464575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.464061    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-bundle" pod "c3c67b89-e2b8-4b5f-a44d-170d24943bbd" (UID: "c3c67b89-e2b8-4b5f-a44d-170d24943bbd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:21.466421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.466387    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.468246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.468219    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-kube-api-access-l2tx8" pod "c3c67b89-e2b8-4b5f-a44d-170d24943bbd" (UID: "c3c67b89-e2b8-4b5f-a44d-170d24943bbd"). InnerVolumeSpecName "kube-api-access-l2tx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:21.471242 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.471219    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-util" pod "c3c67b89-e2b8-4b5f-a44d-170d24943bbd" (UID: "c3c67b89-e2b8-4b5f-a44d-170d24943bbd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:21.495991 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.495971    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:25:21.498879 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.498861    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:25:21.500504 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.500487    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:25:21.501863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.501846    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:25:21.567395 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567341    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8205297a-7593-4712-b393-e1fabfff8a15-kube-api-access-f2t72\" (UniqueName: \"kubernetes.io/projected/8205297a-7593-4712-b393-e1fabfff8a15-kube-api-access-f2t72\") pod \"8205297a-7593-4712-b393-e1fabfff8a15\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") "
Oct 31 18:25:21.567395 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567388    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-bundle\") pod \"66e1f424-62df-4cb4-981d-d356ae245cea\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") "
Oct 31 18:25:21.567580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567423    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-bundle\") pod \"e08f54bd-3381-4b65-95ac-4536eb9cc011\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") "
Oct 31 18:25:21.567580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567450    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-util\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-util\") pod \"66e1f424-62df-4cb4-981d-d356ae245cea\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") "
Oct 31 18:25:21.567580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567495    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-util\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-util\") pod \"8205297a-7593-4712-b393-e1fabfff8a15\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") "
Oct 31 18:25:21.567580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567539    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-util\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-util\") pod \"e08f54bd-3381-4b65-95ac-4536eb9cc011\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") "
Oct 31 18:25:21.567776 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567593    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08f54bd-3381-4b65-95ac-4536eb9cc011-kube-api-access-b5xvj\" (UniqueName: \"kubernetes.io/projected/e08f54bd-3381-4b65-95ac-4536eb9cc011-kube-api-access-b5xvj\") pod \"e08f54bd-3381-4b65-95ac-4536eb9cc011\" (UID: \"e08f54bd-3381-4b65-95ac-4536eb9cc011\") "
Oct 31 18:25:21.567776 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567650    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e1f424-62df-4cb4-981d-d356ae245cea-kube-api-access-lxmcm\" (UniqueName: \"kubernetes.io/projected/66e1f424-62df-4cb4-981d-d356ae245cea-kube-api-access-lxmcm\") pod \"66e1f424-62df-4cb4-981d-d356ae245cea\" (UID: \"66e1f424-62df-4cb4-981d-d356ae245cea\") "
Oct 31 18:25:21.567776 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567679    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-bundle\") pod \"8205297a-7593-4712-b393-e1fabfff8a15\" (UID: \"8205297a-7593-4712-b393-e1fabfff8a15\") "
Oct 31 18:25:21.567923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567860    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-bundle" pod "e08f54bd-3381-4b65-95ac-4536eb9cc011" (UID: "e08f54bd-3381-4b65-95ac-4536eb9cc011"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:21.568159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.567980    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2tx8\" (UniqueName: \"kubernetes.io/projected/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-kube-api-access-l2tx8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.568214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.568171    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.568214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.568187    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3c67b89-e2b8-4b5f-a44d-170d24943bbd-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.568214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.568195    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-bundle" pod "8205297a-7593-4712-b393-e1fabfff8a15" (UID: "8205297a-7593-4712-b393-e1fabfff8a15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:21.568352 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.568227    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-bundle" pod "66e1f424-62df-4cb4-981d-d356ae245cea" (UID: "66e1f424-62df-4cb4-981d-d356ae245cea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:21.570732 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.570690    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08f54bd-3381-4b65-95ac-4536eb9cc011-kube-api-access-b5xvj" pod "e08f54bd-3381-4b65-95ac-4536eb9cc011" (UID: "e08f54bd-3381-4b65-95ac-4536eb9cc011"). InnerVolumeSpecName "kube-api-access-b5xvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:21.570835 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.570783    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e1f424-62df-4cb4-981d-d356ae245cea-kube-api-access-lxmcm" pod "66e1f424-62df-4cb4-981d-d356ae245cea" (UID: "66e1f424-62df-4cb4-981d-d356ae245cea"). InnerVolumeSpecName "kube-api-access-lxmcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:21.570892 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.570869    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8205297a-7593-4712-b393-e1fabfff8a15-kube-api-access-f2t72" pod "8205297a-7593-4712-b393-e1fabfff8a15" (UID: "8205297a-7593-4712-b393-e1fabfff8a15"). InnerVolumeSpecName "kube-api-access-f2t72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:21.573370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.573346    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-util" pod "66e1f424-62df-4cb4-981d-d356ae245cea" (UID: "66e1f424-62df-4cb4-981d-d356ae245cea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:21.574065 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.574044    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-util" pod "e08f54bd-3381-4b65-95ac-4536eb9cc011" (UID: "e08f54bd-3381-4b65-95ac-4536eb9cc011"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:21.575224 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.575202    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-util" pod "8205297a-7593-4712-b393-e1fabfff8a15" (UID: "8205297a-7593-4712-b393-e1fabfff8a15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:21.668915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.668881    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.668915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.668915    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e08f54bd-3381-4b65-95ac-4536eb9cc011-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.669141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.668931    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xvj\" (UniqueName: \"kubernetes.io/projected/e08f54bd-3381-4b65-95ac-4536eb9cc011-kube-api-access-b5xvj\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.669141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.668946    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmcm\" (UniqueName: \"kubernetes.io/projected/66e1f424-62df-4cb4-981d-d356ae245cea-kube-api-access-lxmcm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.669141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.668958    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8205297a-7593-4712-b393-e1fabfff8a15-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.669141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.668971    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2t72\" (UniqueName: \"kubernetes.io/projected/8205297a-7593-4712-b393-e1fabfff8a15-kube-api-access-f2t72\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.669141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.668983    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:21.669141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:21.668997    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66e1f424-62df-4cb4-981d-d356ae245cea-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:22.279990 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.279907    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz" event={"ID":"66e1f424-62df-4cb4-981d-d356ae245cea","Type":"ContainerDied","Data":"889c56e670b92be77f1f3c09da5e0bb7a4ad1058ea8858aeecbbad1f101fb405"}
Oct 31 18:25:22.279990 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.279958    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="889c56e670b92be77f1f3c09da5e0bb7a4ad1058ea8858aeecbbad1f101fb405"
Oct 31 18:25:22.279990 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.279921    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zcmhz"
Oct 31 18:25:22.281938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.281916    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8"
Oct 31 18:25:22.281938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.281925    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fcnf8" event={"ID":"e08f54bd-3381-4b65-95ac-4536eb9cc011","Type":"ContainerDied","Data":"fdc62e5c77ad773cf7f96a96597831f1b8a5a0ae8064884f9ba4c2c88314dd96"}
Oct 31 18:25:22.282129 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.281966    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc62e5c77ad773cf7f96a96597831f1b8a5a0ae8064884f9ba4c2c88314dd96"
Oct 31 18:25:22.283222 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.283202    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:25:22.284007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.283987    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq"
Oct 31 18:25:22.284110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.284007    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq" event={"ID":"8205297a-7593-4712-b393-e1fabfff8a15","Type":"ContainerDied","Data":"8de3512c50551afe15c9b7c16e19595608d239b1c82bd6ae736e75d7cb3885d2"}
Oct 31 18:25:22.284110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.284050    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de3512c50551afe15c9b7c16e19595608d239b1c82bd6ae736e75d7cb3885d2"
Oct 31 18:25:22.286033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.286007    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj" event={"ID":"c3c67b89-e2b8-4b5f-a44d-170d24943bbd","Type":"ContainerDied","Data":"30260bf11185972bab22c6088e4888ae34a85d912c3cc6ab5c5200b018e1580f"}
Oct 31 18:25:22.286129 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.286052    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30260bf11185972bab22c6088e4888ae34a85d912c3cc6ab5c5200b018e1580f"
Oct 31 18:25:22.286129 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.286020    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnmqqj"
Oct 31 18:25:22.410357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.410332    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:25:22.474834 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.474810    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-bundle\") pod \"cc19cf95-db91-4751-ad9d-a3c2118f903a\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") "
Oct 31 18:25:22.474972 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.474842    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc19cf95-db91-4751-ad9d-a3c2118f903a-kube-api-access-nzkmf\" (UniqueName: \"kubernetes.io/projected/cc19cf95-db91-4751-ad9d-a3c2118f903a-kube-api-access-nzkmf\") pod \"cc19cf95-db91-4751-ad9d-a3c2118f903a\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") "
Oct 31 18:25:22.474972 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.474912    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-util\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-util\") pod \"cc19cf95-db91-4751-ad9d-a3c2118f903a\" (UID: \"cc19cf95-db91-4751-ad9d-a3c2118f903a\") "
Oct 31 18:25:22.475675 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.475646    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-bundle" pod "cc19cf95-db91-4751-ad9d-a3c2118f903a" (UID: "cc19cf95-db91-4751-ad9d-a3c2118f903a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:22.477006 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.476983    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc19cf95-db91-4751-ad9d-a3c2118f903a-kube-api-access-nzkmf" pod "cc19cf95-db91-4751-ad9d-a3c2118f903a" (UID: "cc19cf95-db91-4751-ad9d-a3c2118f903a"). InnerVolumeSpecName "kube-api-access-nzkmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:25:22.480819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.480798    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-util" pod "cc19cf95-db91-4751-ad9d-a3c2118f903a" (UID: "cc19cf95-db91-4751-ad9d-a3c2118f903a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:25:22.575788 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.575721    5551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-util\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:22.575788 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.575746    5551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc19cf95-db91-4751-ad9d-a3c2118f903a-bundle\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:22.575788 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:22.575760    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzkmf\" (UniqueName: \"kubernetes.io/projected/cc19cf95-db91-4751-ad9d-a3c2118f903a-kube-api-access-nzkmf\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:25:23.152540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.152503    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"]
Oct 31 18:25:23.153205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153179    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerName="pull"
Oct 31 18:25:23.153205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153206    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerName="pull"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153223    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerName="extract"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153236    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerName="extract"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153249    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerName="util"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153256    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerName="util"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153274    5551 cpu_manager.go:414] "containerMap: removing container" podUID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerName="pull"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153281    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerName="pull"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153292    5551 cpu_manager.go:414] "containerMap: removing container" podUID="66e1f424-62df-4cb4-981d-d356ae245cea" containerName="extract"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153298    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e1f424-62df-4cb4-981d-d356ae245cea" containerName="extract"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153328    5551 cpu_manager.go:414] "containerMap: removing container" podUID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerName="pull"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153335    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerName="pull"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153356    5551 cpu_manager.go:414] "containerMap: removing container" podUID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerName="util"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153363    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerName="util"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153378    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerName="pull"
Oct 31 18:25:23.153389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153385    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153401    5551 cpu_manager.go:414] "containerMap: removing container" podUID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153408    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153425    5551 cpu_manager.go:414] "containerMap: removing container" podUID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153432    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153472    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153479    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153490    5551 cpu_manager.go:414] "containerMap: removing container" podUID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153497    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153509    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153516    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153532    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153539    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153550    5551 cpu_manager.go:414] "containerMap: removing container" podUID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153557    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153585    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153592    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153606    5551 cpu_manager.go:414] "containerMap: removing container" podUID="66e1f424-62df-4cb4-981d-d356ae245cea" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153613    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e1f424-62df-4cb4-981d-d356ae245cea" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153635    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153642    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153655    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153671    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153692    5551 cpu_manager.go:414] "containerMap: removing container" podUID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153701    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153718    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153725    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153736    5551 cpu_manager.go:414] "containerMap: removing container" podUID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153743    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153762    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8205297a-7593-4712-b393-e1fabfff8a15" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153768    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8205297a-7593-4712-b393-e1fabfff8a15" containerName="util"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153788    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8205297a-7593-4712-b393-e1fabfff8a15" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153795    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8205297a-7593-4712-b393-e1fabfff8a15" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153806    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8205297a-7593-4712-b393-e1fabfff8a15" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153813    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8205297a-7593-4712-b393-e1fabfff8a15" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153821    5551 cpu_manager.go:414] "containerMap: removing container" podUID="66e1f424-62df-4cb4-981d-d356ae245cea" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153828    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e1f424-62df-4cb4-981d-d356ae245cea" containerName="pull"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153848    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.153855    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154012    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f42e19-9ab6-4302-b4c5-fef0abe42c35" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154025    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e1f424-62df-4cb4-981d-d356ae245cea" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154045    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="be02ed07-a67b-46ca-bfa5-a2f627901b08" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154057    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c25851-ff28-4e0d-b1a9-1c7b976252fc" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154068    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc19cf95-db91-4751-ad9d-a3c2118f903a" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154076    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe13bc5b-c59c-42b1-adc1-c4eb76883d20" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154091    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c67b89-e2b8-4b5f-a44d-170d24943bbd" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154100    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8205297a-7593-4712-b393-e1fabfff8a15" containerName="extract"
Oct 31 18:25:23.154092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.154110    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08f54bd-3381-4b65-95ac-4536eb9cc011" containerName="extract"
Oct 31 18:25:23.159448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.159424    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.217935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.217889    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 18:25:23.218126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.217982    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 18:25:23.218126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.218024    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-k5llp\""
Oct 31 18:25:23.218126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.218051    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 18:25:23.218126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.218098    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 18:25:23.218284 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.218125    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 18:25:23.266110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.266082    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"]
Oct 31 18:25:23.281383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.281361    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.281764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.281405    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.281764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.281427    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmjj\" (UniqueName: \"kubernetes.io/projected/d6092d9e-0d21-416a-9ecf-393473112dbe-kube-api-access-7lmjj\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.281764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.281565    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.281764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.281591    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6092d9e-0d21-416a-9ecf-393473112dbe-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.281764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.281622    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6092d9e-0d21-416a-9ecf-393473112dbe-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.292733 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.292705    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm" event={"ID":"cc19cf95-db91-4751-ad9d-a3c2118f903a","Type":"ContainerDied","Data":"c44573e86291027b4c1a1c856a8b838f8213a7c91c8a760a113b084312da5e0b"}
Oct 31 18:25:23.292845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.292755    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c44573e86291027b4c1a1c856a8b838f8213a7c91c8a760a113b084312da5e0b"
Oct 31 18:25:23.292845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.292720    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpvjtm"
Oct 31 18:25:23.382132 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.382098    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.382132 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.382139    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6092d9e-0d21-416a-9ecf-393473112dbe-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.382302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.382166    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6092d9e-0d21-416a-9ecf-393473112dbe-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.382302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.382212    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.382302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.382249    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.382302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.382274    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmjj\" (UniqueName: \"kubernetes.io/projected/d6092d9e-0d21-416a-9ecf-393473112dbe-kube-api-access-7lmjj\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.382710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.382683    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6092d9e-0d21-416a-9ecf-393473112dbe-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.384769 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.384746    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6092d9e-0d21-416a-9ecf-393473112dbe-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.384860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.384780    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.384860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.384803    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.384969 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.384921    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6092d9e-0d21-416a-9ecf-393473112dbe-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.443654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.443603    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmjj\" (UniqueName: \"kubernetes.io/projected/d6092d9e-0d21-416a-9ecf-393473112dbe-kube-api-access-7lmjj\") pod \"opendatahub-operator-controller-manager-dd8c87985-ckn7t\" (UID: \"d6092d9e-0d21-416a-9ecf-393473112dbe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.473980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.473963    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:23.960401 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:23.960380    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"]
Oct 31 18:25:23.962768 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:25:23.962742    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6092d9e_0d21_416a_9ecf_393473112dbe.slice/crio-4637997fec85db83605ab00c2c45fa8913c8cc67ce280882bf47f9596b1580b6 WatchSource:0}: Error finding container 4637997fec85db83605ab00c2c45fa8913c8cc67ce280882bf47f9596b1580b6: Status 404 returned error can't find the container with id 4637997fec85db83605ab00c2c45fa8913c8cc67ce280882bf47f9596b1580b6
Oct 31 18:25:24.299235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:24.299156    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t" event={"ID":"d6092d9e-0d21-416a-9ecf-393473112dbe","Type":"ContainerStarted","Data":"4637997fec85db83605ab00c2c45fa8913c8cc67ce280882bf47f9596b1580b6"}
Oct 31 18:25:28.414995 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.414967    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"]
Oct 31 18:25:28.421271 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.421245    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"
Oct 31 18:25:28.476339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.476308    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"]
Oct 31 18:25:28.516018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.515994    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 18:25:28.523981 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.523964    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:25:28.524042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.523964    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-6kc59\""
Oct 31 18:25:28.528394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.528374    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzx5z\" (UniqueName: \"kubernetes.io/projected/0df792ce-86eb-4d09-aa9b-fd6abcfc24b2-kube-api-access-qzx5z\") pod \"jobset-operator-674cd59dc5-vttf4\" (UID: \"0df792ce-86eb-4d09-aa9b-fd6abcfc24b2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"
Oct 31 18:25:28.528447 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.528405    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0df792ce-86eb-4d09-aa9b-fd6abcfc24b2-tmp\") pod \"jobset-operator-674cd59dc5-vttf4\" (UID: \"0df792ce-86eb-4d09-aa9b-fd6abcfc24b2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"
Oct 31 18:25:28.629748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.629713    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzx5z\" (UniqueName: \"kubernetes.io/projected/0df792ce-86eb-4d09-aa9b-fd6abcfc24b2-kube-api-access-qzx5z\") pod \"jobset-operator-674cd59dc5-vttf4\" (UID: \"0df792ce-86eb-4d09-aa9b-fd6abcfc24b2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"
Oct 31 18:25:28.629748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.629753    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0df792ce-86eb-4d09-aa9b-fd6abcfc24b2-tmp\") pod \"jobset-operator-674cd59dc5-vttf4\" (UID: \"0df792ce-86eb-4d09-aa9b-fd6abcfc24b2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"
Oct 31 18:25:28.630133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.630117    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0df792ce-86eb-4d09-aa9b-fd6abcfc24b2-tmp\") pod \"jobset-operator-674cd59dc5-vttf4\" (UID: \"0df792ce-86eb-4d09-aa9b-fd6abcfc24b2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"
Oct 31 18:25:28.687138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.687078    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzx5z\" (UniqueName: \"kubernetes.io/projected/0df792ce-86eb-4d09-aa9b-fd6abcfc24b2-kube-api-access-qzx5z\") pod \"jobset-operator-674cd59dc5-vttf4\" (UID: \"0df792ce-86eb-4d09-aa9b-fd6abcfc24b2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"
Oct 31 18:25:28.762638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:28.762614    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"
Oct 31 18:25:29.139371 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:29.137581    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4"]
Oct 31 18:25:29.142916 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:25:29.142887    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0df792ce_86eb_4d09_aa9b_fd6abcfc24b2.slice/crio-c888dfde32e8d63324d3e486454369a133dc5789e65ff8009a4f4db23064a479 WatchSource:0}: Error finding container c888dfde32e8d63324d3e486454369a133dc5789e65ff8009a4f4db23064a479: Status 404 returned error can't find the container with id c888dfde32e8d63324d3e486454369a133dc5789e65ff8009a4f4db23064a479
Oct 31 18:25:29.329101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:29.329009    5551 generic.go:356] "Generic (PLEG): container finished" podID="d6092d9e-0d21-416a-9ecf-393473112dbe" containerID="9cf08b279e174e7d84a63e19fb9aaa2a07d4fd5c564f00d0f2f39dbe780604e0" exitCode=0
Oct 31 18:25:29.329275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:29.329102    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t" event={"ID":"d6092d9e-0d21-416a-9ecf-393473112dbe","Type":"ContainerDied","Data":"9cf08b279e174e7d84a63e19fb9aaa2a07d4fd5c564f00d0f2f39dbe780604e0"}
Oct 31 18:25:29.330581 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:29.330563    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4" event={"ID":"0df792ce-86eb-4d09-aa9b-fd6abcfc24b2","Type":"ContainerStarted","Data":"c888dfde32e8d63324d3e486454369a133dc5789e65ff8009a4f4db23064a479"}
Oct 31 18:25:30.337015 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:30.336969    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t" event={"ID":"d6092d9e-0d21-416a-9ecf-393473112dbe","Type":"ContainerStarted","Data":"daea6f8282336482e188e3c8c2eb4f6e63640b698058ed41208ab9abb6281fbe"}
Oct 31 18:25:30.337532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:30.337075    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:30.520850 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:30.520803    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t" podStartSLOduration=1.826517189 podStartE2EDuration="7.520785481s" totalImagesPullingTime="5.235290447s" totalInitContainerRuntime="458.977845ms" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:30.470215904 +0000 UTC m=+233.701454193" watchObservedRunningTime="2026-10-31 18:25:30.520785481 +0000 UTC m=+233.752023771"
Oct 31 18:25:33.354594 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:33.354562    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4" event={"ID":"0df792ce-86eb-4d09-aa9b-fd6abcfc24b2","Type":"ContainerStarted","Data":"df59f7c7ececdb29481a499d71f0b1626899f075c6462f39b4a82646e8e093f7"}
Oct 31 18:25:33.467646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:33.467589    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vttf4" podStartSLOduration=2.005425984 podStartE2EDuration="5.467574594s" totalImagesPullingTime="3.46214861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:33.467183747 +0000 UTC m=+236.698422036" watchObservedRunningTime="2026-10-31 18:25:33.467574594 +0000 UTC m=+236.698812880"
Oct 31 18:25:36.699115 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:36.699081    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"]
Oct 31 18:25:36.702867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:36.702849    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"
Oct 31 18:25:36.803228 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:36.803191    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqpl8\" (UniqueName: \"kubernetes.io/projected/14e622e5-332a-4304-a9ec-d0feaa7c4002-kube-api-access-xqpl8\") pod \"authorino-operator-7b7bb4bc77-sbwg7\" (UID: \"14e622e5-332a-4304-a9ec-d0feaa7c4002\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"
Oct 31 18:25:36.807127 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:36.807104    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 18:25:36.807235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:36.807164    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 18:25:36.807235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:36.807164    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f275f\""
Oct 31 18:25:36.808987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:36.808969    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"]
Oct 31 18:25:36.904048 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:36.904006    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqpl8\" (UniqueName: \"kubernetes.io/projected/14e622e5-332a-4304-a9ec-d0feaa7c4002-kube-api-access-xqpl8\") pod \"authorino-operator-7b7bb4bc77-sbwg7\" (UID: \"14e622e5-332a-4304-a9ec-d0feaa7c4002\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"
Oct 31 18:25:37.014799 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:37.014713    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 18:25:37.085989 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:37.085965    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 18:25:37.096499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:37.096451    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqpl8\" (UniqueName: \"kubernetes.io/projected/14e622e5-332a-4304-a9ec-d0feaa7c4002-kube-api-access-xqpl8\") pod \"authorino-operator-7b7bb4bc77-sbwg7\" (UID: \"14e622e5-332a-4304-a9ec-d0feaa7c4002\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"
Oct 31 18:25:37.364782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:37.364709    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f275f\""
Oct 31 18:25:37.373631 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:37.373604    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"
Oct 31 18:25:37.762084 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:37.762063    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"]
Oct 31 18:25:37.764440 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:25:37.764410    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e622e5_332a_4304_a9ec_d0feaa7c4002.slice/crio-2a32765a14e2adc64d6e02f6322b6282b385b8fe1e251de6fd20bf780ae543b2 WatchSource:0}: Error finding container 2a32765a14e2adc64d6e02f6322b6282b385b8fe1e251de6fd20bf780ae543b2: Status 404 returned error can't find the container with id 2a32765a14e2adc64d6e02f6322b6282b385b8fe1e251de6fd20bf780ae543b2
Oct 31 18:25:38.378094 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:38.378053    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7" event={"ID":"14e622e5-332a-4304-a9ec-d0feaa7c4002","Type":"ContainerStarted","Data":"2a32765a14e2adc64d6e02f6322b6282b385b8fe1e251de6fd20bf780ae543b2"}
Oct 31 18:25:41.117759 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.117721    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"]
Oct 31 18:25:41.125052 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.125030    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"
Oct 31 18:25:41.183845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.183808    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"]
Oct 31 18:25:41.230740 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.230711    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 18:25:41.230935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.230775    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mqqzs\""
Oct 31 18:25:41.246322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.246279    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpqb\" (UniqueName: \"kubernetes.io/projected/99d1f84b-a604-46c8-83be-3f53ab8c959c-kube-api-access-bzpqb\") pod \"dns-operator-controller-manager-8557cbdb84-zkmz5\" (UID: \"99d1f84b-a604-46c8-83be-3f53ab8c959c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"
Oct 31 18:25:41.344634 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.344600    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-ckn7t"
Oct 31 18:25:41.347472 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.347419    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpqb\" (UniqueName: \"kubernetes.io/projected/99d1f84b-a604-46c8-83be-3f53ab8c959c-kube-api-access-bzpqb\") pod \"dns-operator-controller-manager-8557cbdb84-zkmz5\" (UID: \"99d1f84b-a604-46c8-83be-3f53ab8c959c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"
Oct 31 18:25:41.404883 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.404852    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpqb\" (UniqueName: \"kubernetes.io/projected/99d1f84b-a604-46c8-83be-3f53ab8c959c-kube-api-access-bzpqb\") pod \"dns-operator-controller-manager-8557cbdb84-zkmz5\" (UID: \"99d1f84b-a604-46c8-83be-3f53ab8c959c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"
Oct 31 18:25:41.438621 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.438596    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"
Oct 31 18:25:41.822766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:41.822683    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"]
Oct 31 18:25:41.827712 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:25:41.827682    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d1f84b_a604_46c8_83be_3f53ab8c959c.slice/crio-178eb976464939d460d1663bd2771378c16562a1535135351192ffbec1fb8a09 WatchSource:0}: Error finding container 178eb976464939d460d1663bd2771378c16562a1535135351192ffbec1fb8a09: Status 404 returned error can't find the container with id 178eb976464939d460d1663bd2771378c16562a1535135351192ffbec1fb8a09
Oct 31 18:25:42.399647 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:42.399609    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7" event={"ID":"14e622e5-332a-4304-a9ec-d0feaa7c4002","Type":"ContainerStarted","Data":"67e1e8dd14dcf9ecc185292fc4c0e5d7b94530819958c6e550c2738de982c37d"}
Oct 31 18:25:42.400084 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:42.399729    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"
Oct 31 18:25:42.400919 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:42.400897    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5" event={"ID":"99d1f84b-a604-46c8-83be-3f53ab8c959c","Type":"ContainerStarted","Data":"178eb976464939d460d1663bd2771378c16562a1535135351192ffbec1fb8a09"}
Oct 31 18:25:42.515649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:42.515594    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7" podStartSLOduration=2.369319329 podStartE2EDuration="6.515579157s" totalImagesPullingTime="4.146259828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:42.514793555 +0000 UTC m=+245.746031879" watchObservedRunningTime="2026-10-31 18:25:42.515579157 +0000 UTC m=+245.746817448"
Oct 31 18:25:45.329814 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.329781    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"]
Oct 31 18:25:45.332336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.332320    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:25:45.422638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.422602    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5" event={"ID":"99d1f84b-a604-46c8-83be-3f53ab8c959c","Type":"ContainerStarted","Data":"96237731c780e520aad50462284791fe84bed08a7c62361883e961b0bb5fcb3b"}
Oct 31 18:25:45.422779 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.422705    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"
Oct 31 18:25:45.437851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.437825    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rptjt\""
Oct 31 18:25:45.439498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.439481    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"]
Oct 31 18:25:45.488711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.488681    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dft\" (UniqueName: \"kubernetes.io/projected/2ff429fc-0746-4d6a-aff4-088caec1919e-kube-api-access-x2dft\") pod \"limitador-operator-controller-manager-5778bb4449-hd4jn\" (UID: \"2ff429fc-0746-4d6a-aff4-088caec1919e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:25:45.577785 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.577733    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5" podStartSLOduration=1.844426095 podStartE2EDuration="4.577718894s" totalImagesPullingTime="2.733292799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:45.57629995 +0000 UTC m=+248.807538239" watchObservedRunningTime="2026-10-31 18:25:45.577718894 +0000 UTC m=+248.808957185"
Oct 31 18:25:45.589150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.589084    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dft\" (UniqueName: \"kubernetes.io/projected/2ff429fc-0746-4d6a-aff4-088caec1919e-kube-api-access-x2dft\") pod \"limitador-operator-controller-manager-5778bb4449-hd4jn\" (UID: \"2ff429fc-0746-4d6a-aff4-088caec1919e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:25:45.650973 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.650938    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dft\" (UniqueName: \"kubernetes.io/projected/2ff429fc-0746-4d6a-aff4-088caec1919e-kube-api-access-x2dft\") pod \"limitador-operator-controller-manager-5778bb4449-hd4jn\" (UID: \"2ff429fc-0746-4d6a-aff4-088caec1919e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:25:45.943671 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:45.943625    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:25:46.379486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:46.379445    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"]
Oct 31 18:25:46.382165 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:25:46.382140    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff429fc_0746_4d6a_aff4_088caec1919e.slice/crio-4cd3a9f8c5cbd95d41d8cea3483c9a39b8a132377e0da00ddf56d9c491ba3a61 WatchSource:0}: Error finding container 4cd3a9f8c5cbd95d41d8cea3483c9a39b8a132377e0da00ddf56d9c491ba3a61: Status 404 returned error can't find the container with id 4cd3a9f8c5cbd95d41d8cea3483c9a39b8a132377e0da00ddf56d9c491ba3a61
Oct 31 18:25:46.432869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:46.432837    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn" event={"ID":"2ff429fc-0746-4d6a-aff4-088caec1919e","Type":"ContainerStarted","Data":"4cd3a9f8c5cbd95d41d8cea3483c9a39b8a132377e0da00ddf56d9c491ba3a61"}
Oct 31 18:25:46.603163 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:25:46.603125    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:25:50.455519 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:50.455471    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn" event={"ID":"2ff429fc-0746-4d6a-aff4-088caec1919e","Type":"ContainerStarted","Data":"d5b0a4c4983b4b1d4a5d6d1c9fab26676cfd5692eec0b97632e291893b42477f"}
Oct 31 18:25:50.455931 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:50.455561    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:25:50.579024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:50.578967    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn" podStartSLOduration=2.043744246 podStartE2EDuration="5.578953542s" totalImagesPullingTime="3.535209296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:50.577645698 +0000 UTC m=+253.808883987" watchObservedRunningTime="2026-10-31 18:25:50.578953542 +0000 UTC m=+253.810191831"
Oct 31 18:25:51.243008 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.242976    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"]
Oct 31 18:25:51.246587 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.246568    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:25:51.302175 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.302143    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"]
Oct 31 18:25:51.345767 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.345734    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7l2g\" (UniqueName: \"kubernetes.io/projected/e8c99ace-8a40-41aa-9615-809def49b98f-kube-api-access-q7l2g\") pod \"kuadrant-operator-controller-manager-678d87766f-q2zdr\" (UID: \"e8c99ace-8a40-41aa-9615-809def49b98f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:25:51.345899 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.345798    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8c99ace-8a40-41aa-9615-809def49b98f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q2zdr\" (UID: \"e8c99ace-8a40-41aa-9615-809def49b98f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:25:51.349025 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.349010    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6tbfg\""
Oct 31 18:25:51.446193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.446162    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7l2g\" (UniqueName: \"kubernetes.io/projected/e8c99ace-8a40-41aa-9615-809def49b98f-kube-api-access-q7l2g\") pod \"kuadrant-operator-controller-manager-678d87766f-q2zdr\" (UID: \"e8c99ace-8a40-41aa-9615-809def49b98f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:25:51.446325 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.446251    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8c99ace-8a40-41aa-9615-809def49b98f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q2zdr\" (UID: \"e8c99ace-8a40-41aa-9615-809def49b98f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:25:51.446666 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.446645    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8c99ace-8a40-41aa-9615-809def49b98f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q2zdr\" (UID: \"e8c99ace-8a40-41aa-9615-809def49b98f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:25:51.502659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.502606    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7l2g\" (UniqueName: \"kubernetes.io/projected/e8c99ace-8a40-41aa-9615-809def49b98f-kube-api-access-q7l2g\") pod \"kuadrant-operator-controller-manager-678d87766f-q2zdr\" (UID: \"e8c99ace-8a40-41aa-9615-809def49b98f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:25:51.559082 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:51.559057    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:25:52.035765 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:52.035738    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"]
Oct 31 18:25:52.038941 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:25:52.038918    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c99ace_8a40_41aa_9615_809def49b98f.slice/crio-0316e25ae5bd4ee23db657767f8d1c9244bbf3bb4aeb927ca987cdfdb0aed27c WatchSource:0}: Error finding container 0316e25ae5bd4ee23db657767f8d1c9244bbf3bb4aeb927ca987cdfdb0aed27c: Status 404 returned error can't find the container with id 0316e25ae5bd4ee23db657767f8d1c9244bbf3bb4aeb927ca987cdfdb0aed27c
Oct 31 18:25:52.465934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:52.465897    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr" event={"ID":"e8c99ace-8a40-41aa-9615-809def49b98f","Type":"ContainerStarted","Data":"0316e25ae5bd4ee23db657767f8d1c9244bbf3bb4aeb927ca987cdfdb0aed27c"}
Oct 31 18:25:53.413944 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:53.413917    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sbwg7"
Oct 31 18:25:56.435023 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:25:56.434994    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zkmz5"
Oct 31 18:26:00.261698 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.261663    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"]
Oct 31 18:26:00.267938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.267911    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317496    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317551    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317640    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bs6d\" (UniqueName: \"kubernetes.io/projected/42ddf73a-f3d8-46de-b88f-a5d887e21df8-kube-api-access-5bs6d\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317692    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317719    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317924 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317748    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317924 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317832    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317924 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317876    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.317924 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.317913    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.345719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.345691    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hls89\""
Oct 31 18:26:00.371790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.371756    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"]
Oct 31 18:26:00.418694 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418662    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.418896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418706    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.418896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418750    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.418896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418786    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.418896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418818    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bs6d\" (UniqueName: \"kubernetes.io/projected/42ddf73a-f3d8-46de-b88f-a5d887e21df8-kube-api-access-5bs6d\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.418896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418851    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.418896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418877    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.419190 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418910    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.419190 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.418934    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.419362 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.419202    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.419362 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.419291    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.419362 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.419329    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.419534 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.419520    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.419748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.419714    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.421239 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.421216    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.421585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.421567    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.476066 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.476036    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddf73a-f3d8-46de-b88f-a5d887e21df8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.476206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.476186    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bs6d\" (UniqueName: \"kubernetes.io/projected/42ddf73a-f3d8-46de-b88f-a5d887e21df8-kube-api-access-5bs6d\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq\" (UID: \"42ddf73a-f3d8-46de-b88f-a5d887e21df8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:00.580615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:00.580544    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:01.009156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.009119    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"]
Oct 31 18:26:01.049114 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:26:01.049086    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ddf73a_f3d8_46de_b88f_a5d887e21df8.slice/crio-47547cfadd40ef80169f37237e491d0afb573a44e651797c93680e3098ec80e0 WatchSource:0}: Error finding container 47547cfadd40ef80169f37237e491d0afb573a44e651797c93680e3098ec80e0: Status 404 returned error can't find the container with id 47547cfadd40ef80169f37237e491d0afb573a44e651797c93680e3098ec80e0
Oct 31 18:26:01.051260 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.051226    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:26:01.051367 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.051307    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:26:01.051367 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.051351    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:26:01.462529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.462430    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:26:01.512743 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.512708    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq" event={"ID":"42ddf73a-f3d8-46de-b88f-a5d887e21df8","Type":"ContainerStarted","Data":"2f72522380322984da2ba7f44d79f30baa76823bf57d08beb3419c5a42dda518"}
Oct 31 18:26:01.512898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.512786    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq" event={"ID":"42ddf73a-f3d8-46de-b88f-a5d887e21df8","Type":"ContainerStarted","Data":"47547cfadd40ef80169f37237e491d0afb573a44e651797c93680e3098ec80e0"}
Oct 31 18:26:01.580888 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.580861    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:01.691143 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:01.691093    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq" podStartSLOduration=1.6910776269999999 podStartE2EDuration="1.691077627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:01.689649787 +0000 UTC m=+264.920888076" watchObservedRunningTime="2026-10-31 18:26:01.691077627 +0000 UTC m=+264.922315916"
Oct 31 18:26:02.585986 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:02.585951    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:03.523915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:03.523884    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:03.525031 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:03.524997    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq"
Oct 31 18:26:05.536681 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:05.536644    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr" event={"ID":"e8c99ace-8a40-41aa-9615-809def49b98f","Type":"ContainerStarted","Data":"74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3"}
Oct 31 18:26:05.537174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:05.536794    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:26:05.754654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:05.754604    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr" podStartSLOduration=1.751687534 podStartE2EDuration="14.754588565s" totalImagesPullingTime="13.002901031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:05.702607026 +0000 UTC m=+268.933845327" watchObservedRunningTime="2026-10-31 18:26:05.754588565 +0000 UTC m=+268.985826854"
Oct 31 18:26:09.048995 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.048962    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2n4kw"]
Oct 31 18:26:09.052598 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.052581    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2n4kw"
Oct 31 18:26:09.099174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.099146    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5s4\" (UniqueName: \"kubernetes.io/projected/6be95202-0cd8-460c-a124-d114c7c0b0bb-kube-api-access-gw5s4\") pod \"authorino-57f7f8659f-2n4kw\" (UID: \"6be95202-0cd8-460c-a124-d114c7c0b0bb\") " pod="kuadrant-system/authorino-57f7f8659f-2n4kw"
Oct 31 18:26:09.105275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.105253    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2n4kw"]
Oct 31 18:26:09.152752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.152728    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rxghs\""
Oct 31 18:26:09.200611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.200585    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5s4\" (UniqueName: \"kubernetes.io/projected/6be95202-0cd8-460c-a124-d114c7c0b0bb-kube-api-access-gw5s4\") pod \"authorino-57f7f8659f-2n4kw\" (UID: \"6be95202-0cd8-460c-a124-d114c7c0b0bb\") " pod="kuadrant-system/authorino-57f7f8659f-2n4kw"
Oct 31 18:26:09.256987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.256959    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5s4\" (UniqueName: \"kubernetes.io/projected/6be95202-0cd8-460c-a124-d114c7c0b0bb-kube-api-access-gw5s4\") pod \"authorino-57f7f8659f-2n4kw\" (UID: \"6be95202-0cd8-460c-a124-d114c7c0b0bb\") " pod="kuadrant-system/authorino-57f7f8659f-2n4kw"
Oct 31 18:26:09.364211 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.364134    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2n4kw"
Oct 31 18:26:09.735955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:09.735812    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2n4kw"]
Oct 31 18:26:09.738227 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:26:09.738190    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6be95202_0cd8_460c_a124_d114c7c0b0bb.slice/crio-35ca0aabbde6530f2d8a67f381f37236b4f9e1d04f3c90e743021cf15ba702f2 WatchSource:0}: Error finding container 35ca0aabbde6530f2d8a67f381f37236b4f9e1d04f3c90e743021cf15ba702f2: Status 404 returned error can't find the container with id 35ca0aabbde6530f2d8a67f381f37236b4f9e1d04f3c90e743021cf15ba702f2
Oct 31 18:26:10.563807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:10.563774    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2n4kw" event={"ID":"6be95202-0cd8-460c-a124-d114c7c0b0bb","Type":"ContainerStarted","Data":"35ca0aabbde6530f2d8a67f381f37236b4f9e1d04f3c90e743021cf15ba702f2"}
Oct 31 18:26:11.844256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.844223    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"]
Oct 31 18:26:11.848054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.848032    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:11.856637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.856592    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j5l26"]
Oct 31 18:26:11.860151 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.860130    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26"
Oct 31 18:26:11.900026 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.900002    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 18:26:11.903444 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.903423    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 18:26:11.903444 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.903426    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-bccf4\""
Oct 31 18:26:11.927072 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.927042    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfc5\" (UniqueName: \"kubernetes.io/projected/58ef3323-2763-4091-8d61-f4600a97ad9b-kube-api-access-kqfc5\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:11.927208 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.927096    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58ef3323-2763-4091-8d61-f4600a97ad9b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:11.927257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.927222    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/58ef3323-2763-4091-8d61-f4600a97ad9b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:11.927307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.927254    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7c6x\" (UniqueName: \"kubernetes.io/projected/05aaf52b-dcc8-4a4e-87c1-e211b979dc51-kube-api-access-x7c6x\") pod \"authorino-56c6c9ddd5-j5l26\" (UID: \"05aaf52b-dcc8-4a4e-87c1-e211b979dc51\") " pod="kuadrant-system/authorino-56c6c9ddd5-j5l26"
Oct 31 18:26:11.927369 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:11.927315    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58ef3323-2763-4091-8d61-f4600a97ad9b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.028752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.028715    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfc5\" (UniqueName: \"kubernetes.io/projected/58ef3323-2763-4091-8d61-f4600a97ad9b-kube-api-access-kqfc5\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.028752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.028764    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58ef3323-2763-4091-8d61-f4600a97ad9b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.028986 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.028903    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/58ef3323-2763-4091-8d61-f4600a97ad9b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.028986 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.028957    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7c6x\" (UniqueName: \"kubernetes.io/projected/05aaf52b-dcc8-4a4e-87c1-e211b979dc51-kube-api-access-x7c6x\") pod \"authorino-56c6c9ddd5-j5l26\" (UID: \"05aaf52b-dcc8-4a4e-87c1-e211b979dc51\") " pod="kuadrant-system/authorino-56c6c9ddd5-j5l26"
Oct 31 18:26:12.029077 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.029025    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58ef3323-2763-4091-8d61-f4600a97ad9b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.029263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.029245    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/58ef3323-2763-4091-8d61-f4600a97ad9b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.029297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.029280    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58ef3323-2763-4091-8d61-f4600a97ad9b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.049957 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.049922    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 31 18:26:12.052215 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.052176    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j5l26"]
Oct 31 18:26:12.054176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.054149    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"]
Oct 31 18:26:12.059840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.059813    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58ef3323-2763-4091-8d61-f4600a97ad9b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.086485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.086429    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7c6x\" (UniqueName: \"kubernetes.io/projected/05aaf52b-dcc8-4a4e-87c1-e211b979dc51-kube-api-access-x7c6x\") pod \"authorino-56c6c9ddd5-j5l26\" (UID: \"05aaf52b-dcc8-4a4e-87c1-e211b979dc51\") " pod="kuadrant-system/authorino-56c6c9ddd5-j5l26"
Oct 31 18:26:12.086720 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.086699    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfc5\" (UniqueName: \"kubernetes.io/projected/58ef3323-2763-4091-8d61-f4600a97ad9b-kube-api-access-kqfc5\") pod \"kserve-module-controller-manager-7566cf5fc8-sr4rh\" (UID: \"58ef3323-2763-4091-8d61-f4600a97ad9b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.163156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.163119    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:12.171408 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.171378    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26"
Oct 31 18:26:12.595074 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.595012    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j5l26"]
Oct 31 18:26:12.597936 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:12.597909    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"]
Oct 31 18:26:12.598184 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:26:12.598106    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05aaf52b_dcc8_4a4e_87c1_e211b979dc51.slice/crio-6c4dc12f8a396aae1b8e651ddbfa9b1ba16a53aeed03cd50ed02518684bfe55f WatchSource:0}: Error finding container 6c4dc12f8a396aae1b8e651ddbfa9b1ba16a53aeed03cd50ed02518684bfe55f: Status 404 returned error can't find the container with id 6c4dc12f8a396aae1b8e651ddbfa9b1ba16a53aeed03cd50ed02518684bfe55f
Oct 31 18:26:12.601199 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:26:12.601172    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ef3323_2763_4091_8d61_f4600a97ad9b.slice/crio-5540eedd4cb410de928787a41977371640a721ce230b105d1f13b7952e439c7f WatchSource:0}: Error finding container 5540eedd4cb410de928787a41977371640a721ce230b105d1f13b7952e439c7f: Status 404 returned error can't find the container with id 5540eedd4cb410de928787a41977371640a721ce230b105d1f13b7952e439c7f
Oct 31 18:26:13.580744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:13.580706    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26" event={"ID":"05aaf52b-dcc8-4a4e-87c1-e211b979dc51","Type":"ContainerStarted","Data":"6c4dc12f8a396aae1b8e651ddbfa9b1ba16a53aeed03cd50ed02518684bfe55f"}
Oct 31 18:26:13.582237 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:13.582203    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2n4kw" event={"ID":"6be95202-0cd8-460c-a124-d114c7c0b0bb","Type":"ContainerStarted","Data":"0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7"}
Oct 31 18:26:13.583362 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:13.583340    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh" event={"ID":"58ef3323-2763-4091-8d61-f4600a97ad9b","Type":"ContainerStarted","Data":"5540eedd4cb410de928787a41977371640a721ce230b105d1f13b7952e439c7f"}
Oct 31 18:26:13.704792 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:13.704715    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2n4kw" podStartSLOduration=1.405963167 podStartE2EDuration="4.704697513s" totalImagesPullingTime="3.298734346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:13.703242854 +0000 UTC m=+276.934481135" watchObservedRunningTime="2026-10-31 18:26:13.704697513 +0000 UTC m=+276.935935802"
Oct 31 18:26:14.590214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:14.590139    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26" event={"ID":"05aaf52b-dcc8-4a4e-87c1-e211b979dc51","Type":"ContainerStarted","Data":"ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71"}
Oct 31 18:26:14.702385 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:14.702335    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26" podStartSLOduration=2.064096201 podStartE2EDuration="3.70232101s" totalImagesPullingTime="1.638224809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:14.701383664 +0000 UTC m=+277.932621952" watchObservedRunningTime="2026-10-31 18:26:14.70232101 +0000 UTC m=+277.933559299"
Oct 31 18:26:14.729648 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:14.729618    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2n4kw"]
Oct 31 18:26:15.595973 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:15.595941    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2n4kw" podUID="6be95202-0cd8-460c-a124-d114c7c0b0bb" containerName="authorino" containerID="cri-o://0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7" gracePeriod=30
Oct 31 18:26:15.837505 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:15.837476    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2n4kw"
Oct 31 18:26:15.969118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:15.969084    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be95202-0cd8-460c-a124-d114c7c0b0bb-kube-api-access-gw5s4\" (UniqueName: \"kubernetes.io/projected/6be95202-0cd8-460c-a124-d114c7c0b0bb-kube-api-access-gw5s4\") pod \"6be95202-0cd8-460c-a124-d114c7c0b0bb\" (UID: \"6be95202-0cd8-460c-a124-d114c7c0b0bb\") "
Oct 31 18:26:15.972017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:15.971986    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be95202-0cd8-460c-a124-d114c7c0b0bb-kube-api-access-gw5s4" pod "6be95202-0cd8-460c-a124-d114c7c0b0bb" (UID: "6be95202-0cd8-460c-a124-d114c7c0b0bb"). InnerVolumeSpecName "kube-api-access-gw5s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:26:16.070578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.070539    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5s4\" (UniqueName: \"kubernetes.io/projected/6be95202-0cd8-460c-a124-d114c7c0b0bb-kube-api-access-gw5s4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:26:16.545316 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.545238    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 18:26:16.602958 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.602923    5551 generic.go:356] "Generic (PLEG): container finished" podID="6be95202-0cd8-460c-a124-d114c7c0b0bb" containerID="0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7" exitCode=0
Oct 31 18:26:16.603382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.603009    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2n4kw"
Oct 31 18:26:16.603382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.602993    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2n4kw" event={"ID":"6be95202-0cd8-460c-a124-d114c7c0b0bb","Type":"ContainerDied","Data":"0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7"}
Oct 31 18:26:16.603382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.603220    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2n4kw" event={"ID":"6be95202-0cd8-460c-a124-d114c7c0b0bb","Type":"ContainerDied","Data":"35ca0aabbde6530f2d8a67f381f37236b4f9e1d04f3c90e743021cf15ba702f2"}
Oct 31 18:26:16.603382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.603251    5551 scope.go:122] "RemoveContainer" containerID="0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7"
Oct 31 18:26:16.603999 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:26:16.603974    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:26:16.604873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.604852    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh" event={"ID":"58ef3323-2763-4091-8d61-f4600a97ad9b","Type":"ContainerStarted","Data":"a4c38ed94e0cd0513a377d98d0bfbcd74395d05c2c144a5d3c60e16e4f68d592"}
Oct 31 18:26:16.605005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.604993    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:16.612670 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.612639    5551 scope.go:122] "RemoveContainer" containerID="0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7"
Oct 31 18:26:16.612928 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:26:16.612911    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7\": container with ID starting with 0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7 not found: ID does not exist" containerID="0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7"
Oct 31 18:26:16.612978 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.612938    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7"} err="failed to get container status \"0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7\": rpc error: code = NotFound desc = could not find container \"0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7\": container with ID starting with 0e20f462a3f5e65c75b56de95b8f18d78c39b2680c7854994f478c23646cfcc7 not found: ID does not exist"
Oct 31 18:26:16.825653 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.825582    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2n4kw"]
Oct 31 18:26:16.827415 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.827393    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2n4kw"]
Oct 31 18:26:16.907341 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.907307    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be95202-0cd8-460c-a124-d114c7c0b0bb" path="/var/lib/kubelet/pods/6be95202-0cd8-460c-a124-d114c7c0b0bb/volumes"
Oct 31 18:26:16.945007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:16.944948    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh" podStartSLOduration=2.382892456 podStartE2EDuration="5.944930304s" totalImagesPullingTime="3.562037848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:16.943770123 +0000 UTC m=+280.175008450" watchObservedRunningTime="2026-10-31 18:26:16.944930304 +0000 UTC m=+280.176168596"
Oct 31 18:26:21.907645 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:21.907602    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"]
Oct 31 18:26:21.908246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:21.908233    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6be95202-0cd8-460c-a124-d114c7c0b0bb" containerName="authorino"
Oct 31 18:26:21.908319 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:21.908252    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be95202-0cd8-460c-a124-d114c7c0b0bb" containerName="authorino"
Oct 31 18:26:21.908374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:21.908358    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be95202-0cd8-460c-a124-d114c7c0b0bb" containerName="authorino"
Oct 31 18:26:21.913564 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:21.913538    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:21.967609 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:21.967586    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"]
Oct 31 18:26:22.015243 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.015193    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 18:26:22.015774 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.015758    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 18:26:22.015774 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.015767    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-nkfbj\""
Oct 31 18:26:22.015869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.015772    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 18:26:22.023164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.023134    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7685af-3063-4c11-81c2-78c221aafd89-metrics-certs\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.023235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.023182    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a7685af-3063-4c11-81c2-78c221aafd89-cert\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.023235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.023221    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zfl\" (UniqueName: \"kubernetes.io/projected/6a7685af-3063-4c11-81c2-78c221aafd89-kube-api-access-b4zfl\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.023347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.023336    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a7685af-3063-4c11-81c2-78c221aafd89-manager-config\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.124411 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.124378    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7685af-3063-4c11-81c2-78c221aafd89-metrics-certs\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.124606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.124420    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a7685af-3063-4c11-81c2-78c221aafd89-cert\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.124606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.124495    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zfl\" (UniqueName: \"kubernetes.io/projected/6a7685af-3063-4c11-81c2-78c221aafd89-kube-api-access-b4zfl\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.124606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.124528    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a7685af-3063-4c11-81c2-78c221aafd89-manager-config\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.125219 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.125195    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a7685af-3063-4c11-81c2-78c221aafd89-manager-config\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.127111 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.127090    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a7685af-3063-4c11-81c2-78c221aafd89-cert\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.127199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.127145    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7685af-3063-4c11-81c2-78c221aafd89-metrics-certs\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.180787 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.180734    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zfl\" (UniqueName: \"kubernetes.io/projected/6a7685af-3063-4c11-81c2-78c221aafd89-kube-api-access-b4zfl\") pod \"jobset-controller-manager-596bcdfcd6-rhglw\" (UID: \"6a7685af-3063-4c11-81c2-78c221aafd89\") " pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.225934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.225910    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:22.597048 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.597027    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"]
Oct 31 18:26:22.598890 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:26:22.598860    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7685af_3063_4c11_81c2_78c221aafd89.slice/crio-7ad2ed74d96d69229106b4b133775808bc50a96dd67b1179f46bed953dedcabc WatchSource:0}: Error finding container 7ad2ed74d96d69229106b4b133775808bc50a96dd67b1179f46bed953dedcabc: Status 404 returned error can't find the container with id 7ad2ed74d96d69229106b4b133775808bc50a96dd67b1179f46bed953dedcabc
Oct 31 18:26:22.640816 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:22.640783    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw" event={"ID":"6a7685af-3063-4c11-81c2-78c221aafd89","Type":"ContainerStarted","Data":"7ad2ed74d96d69229106b4b133775808bc50a96dd67b1179f46bed953dedcabc"}
Oct 31 18:26:29.686358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:29.686321    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw" event={"ID":"6a7685af-3063-4c11-81c2-78c221aafd89","Type":"ContainerStarted","Data":"692d58b8429d492e0f78aae7a298db9846437849d865ca98ea8bc6cd9acaf03a"}
Oct 31 18:26:29.686825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:29.686506    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:29.801826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:29.801773    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw" podStartSLOduration=2.539490065 podStartE2EDuration="8.801761335s" totalImagesPullingTime="6.26227127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:29.800368773 +0000 UTC m=+293.031607063" watchObservedRunningTime="2026-10-31 18:26:29.801761335 +0000 UTC m=+293.032999624"
Oct 31 18:26:36.931554 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:36.931524    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:26:36.931956 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:36.931836    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:26:37.613137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:37.613104    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sr4rh"
Oct 31 18:26:40.696897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:26:40.696868    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-596bcdfcd6-rhglw"
Oct 31 18:26:46.604410 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:26:46.604363    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:27:16.603140 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:27:16.603106    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:27:46.604398 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:27:46.604290    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:27:49.980218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:49.980187    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-k7smf"]
Oct 31 18:27:49.983833 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:49.983809    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-k7smf"
Oct 31 18:27:50.040134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.040112    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-k7smf"]
Oct 31 18:27:50.136958 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.136928    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56pk\" (UniqueName: \"kubernetes.io/projected/5a618f2d-f0a8-45aa-9374-2ee554bac60f-kube-api-access-g56pk\") pod \"authorino-6f6944bcdd-k7smf\" (UID: \"5a618f2d-f0a8-45aa-9374-2ee554bac60f\") " pod="kuadrant-system/authorino-6f6944bcdd-k7smf"
Oct 31 18:27:50.238479 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.238359    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56pk\" (UniqueName: \"kubernetes.io/projected/5a618f2d-f0a8-45aa-9374-2ee554bac60f-kube-api-access-g56pk\") pod \"authorino-6f6944bcdd-k7smf\" (UID: \"5a618f2d-f0a8-45aa-9374-2ee554bac60f\") " pod="kuadrant-system/authorino-6f6944bcdd-k7smf"
Oct 31 18:27:50.297144 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.297114    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56pk\" (UniqueName: \"kubernetes.io/projected/5a618f2d-f0a8-45aa-9374-2ee554bac60f-kube-api-access-g56pk\") pod \"authorino-6f6944bcdd-k7smf\" (UID: \"5a618f2d-f0a8-45aa-9374-2ee554bac60f\") " pod="kuadrant-system/authorino-6f6944bcdd-k7smf"
Oct 31 18:27:50.346710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.346677    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-k7smf"]
Oct 31 18:27:50.346963 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.346946    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-k7smf"
Oct 31 18:27:50.371776 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.371748    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58b675c9d7-9zlvm"]
Oct 31 18:27:50.376139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.376116    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b675c9d7-9zlvm"
Oct 31 18:27:50.429768 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.429718    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58b675c9d7-9zlvm"]
Oct 31 18:27:50.541710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.541615    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxfkl\" (UniqueName: \"kubernetes.io/projected/027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216-kube-api-access-pxfkl\") pod \"authorino-58b675c9d7-9zlvm\" (UID: \"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216\") " pod="kuadrant-system/authorino-58b675c9d7-9zlvm"
Oct 31 18:27:50.642950 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.642910    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfkl\" (UniqueName: \"kubernetes.io/projected/027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216-kube-api-access-pxfkl\") pod \"authorino-58b675c9d7-9zlvm\" (UID: \"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216\") " pod="kuadrant-system/authorino-58b675c9d7-9zlvm"
Oct 31 18:27:50.699275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.699247    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfkl\" (UniqueName: \"kubernetes.io/projected/027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216-kube-api-access-pxfkl\") pod \"authorino-58b675c9d7-9zlvm\" (UID: \"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216\") " pod="kuadrant-system/authorino-58b675c9d7-9zlvm"
Oct 31 18:27:50.719663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.719634    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-k7smf"]
Oct 31 18:27:50.722344 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:27:50.722322    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a618f2d_f0a8_45aa_9374_2ee554bac60f.slice/crio-0d770b4db2042b0dbbea078bd60e8012873b16cade99aa90367d0a4ecd8b3efa WatchSource:0}: Error finding container 0d770b4db2042b0dbbea078bd60e8012873b16cade99aa90367d0a4ecd8b3efa: Status 404 returned error can't find the container with id 0d770b4db2042b0dbbea078bd60e8012873b16cade99aa90367d0a4ecd8b3efa
Oct 31 18:27:50.723660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.723639    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:27:50.789613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.789588    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58b675c9d7-9zlvm"]
Oct 31 18:27:50.789797 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.789783    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b675c9d7-9zlvm"
Oct 31 18:27:50.818663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.818629    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dc886c4c7-ppfbh"]
Oct 31 18:27:50.823060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.823029    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:27:50.876054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.876012    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 18:27:50.878208 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.878184    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dc886c4c7-ppfbh"]
Oct 31 18:27:50.945412 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.945377    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/76ff8676-9921-458f-9081-1ea97cf63a31-tls-cert\") pod \"authorino-7dc886c4c7-ppfbh\" (UID: \"76ff8676-9921-458f-9081-1ea97cf63a31\") " pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:27:50.945640 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:50.945609    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9zg\" (UniqueName: \"kubernetes.io/projected/76ff8676-9921-458f-9081-1ea97cf63a31-kube-api-access-dl9zg\") pod \"authorino-7dc886c4c7-ppfbh\" (UID: \"76ff8676-9921-458f-9081-1ea97cf63a31\") " pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:27:51.046827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:51.046737    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9zg\" (UniqueName: \"kubernetes.io/projected/76ff8676-9921-458f-9081-1ea97cf63a31-kube-api-access-dl9zg\") pod \"authorino-7dc886c4c7-ppfbh\" (UID: \"76ff8676-9921-458f-9081-1ea97cf63a31\") " pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:27:51.046827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:51.046801    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/76ff8676-9921-458f-9081-1ea97cf63a31-tls-cert\") pod \"authorino-7dc886c4c7-ppfbh\" (UID: \"76ff8676-9921-458f-9081-1ea97cf63a31\") " pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:27:51.049752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:51.049723    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/76ff8676-9921-458f-9081-1ea97cf63a31-tls-cert\") pod \"authorino-7dc886c4c7-ppfbh\" (UID: \"76ff8676-9921-458f-9081-1ea97cf63a31\") " pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:27:51.104083 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:51.104049    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9zg\" (UniqueName: \"kubernetes.io/projected/76ff8676-9921-458f-9081-1ea97cf63a31-kube-api-access-dl9zg\") pod \"authorino-7dc886c4c7-ppfbh\" (UID: \"76ff8676-9921-458f-9081-1ea97cf63a31\") " pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:27:51.127185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:51.127158    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-k7smf" event={"ID":"5a618f2d-f0a8-45aa-9374-2ee554bac60f","Type":"ContainerStarted","Data":"0d770b4db2042b0dbbea078bd60e8012873b16cade99aa90367d0a4ecd8b3efa"}
Oct 31 18:27:51.138325 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:51.138307    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:27:51.165649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:51.165622    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58b675c9d7-9zlvm"]
Oct 31 18:27:51.169416 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:27:51.169382    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027d1bb7_6bbf_4eb9_a2b7_d7eecf6b0216.slice/crio-c99bafbf146bd4280ed495e11ef3bd6bf45ba34c4f927b168ca940e7491d0e09 WatchSource:0}: Error finding container c99bafbf146bd4280ed495e11ef3bd6bf45ba34c4f927b168ca940e7491d0e09: Status 404 returned error can't find the container with id c99bafbf146bd4280ed495e11ef3bd6bf45ba34c4f927b168ca940e7491d0e09
Oct 31 18:27:51.506751 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:51.506729    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dc886c4c7-ppfbh"]
Oct 31 18:27:51.508685 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:27:51.508660    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76ff8676_9921_458f_9081_1ea97cf63a31.slice/crio-1b5dec908f6cbf6dc56dbb6da2d7fc3e52bf262f32409e0b9162a18e7497b5e7 WatchSource:0}: Error finding container 1b5dec908f6cbf6dc56dbb6da2d7fc3e52bf262f32409e0b9162a18e7497b5e7: Status 404 returned error can't find the container with id 1b5dec908f6cbf6dc56dbb6da2d7fc3e52bf262f32409e0b9162a18e7497b5e7
Oct 31 18:27:52.133592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:52.133552    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b675c9d7-9zlvm" event={"ID":"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216","Type":"ContainerStarted","Data":"c99bafbf146bd4280ed495e11ef3bd6bf45ba34c4f927b168ca940e7491d0e09"}
Oct 31 18:27:52.134828 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:52.134798    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh" event={"ID":"76ff8676-9921-458f-9081-1ea97cf63a31","Type":"ContainerStarted","Data":"1b5dec908f6cbf6dc56dbb6da2d7fc3e52bf262f32409e0b9162a18e7497b5e7"}
Oct 31 18:27:52.139957 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:52.139930    5551 generic.go:356] "Generic (PLEG): container finished" podID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerID="ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780" exitCode=0
Oct 31 18:27:52.140068 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:52.139978    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" event={"ID":"9525d3e3-170d-42d1-a7cf-89a87b80d0fc","Type":"ContainerDied","Data":"ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780"}
Oct 31 18:27:52.140379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:52.140364    5551 scope.go:122] "RemoveContainer" containerID="ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780"
Oct 31 18:27:52.141613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:52.141587    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:27:52.141695 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:52.141659    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:27:52.141733 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:52.141693    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:27:53.179728 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.179687    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b675c9d7-9zlvm" event={"ID":"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216","Type":"ContainerStarted","Data":"6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4"}
Oct 31 18:27:53.180501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.180013    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-58b675c9d7-9zlvm" podUID="027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216" containerName="authorino" containerID="cri-o://6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4" gracePeriod=30
Oct 31 18:27:53.187410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.187355    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh" event={"ID":"76ff8676-9921-458f-9081-1ea97cf63a31","Type":"ContainerStarted","Data":"3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e"}
Oct 31 18:27:53.190421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.190393    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-k7smf" event={"ID":"5a618f2d-f0a8-45aa-9374-2ee554bac60f","Type":"ContainerStarted","Data":"05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e"}
Oct 31 18:27:53.190584 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.190560    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-k7smf" podUID="5a618f2d-f0a8-45aa-9374-2ee554bac60f" containerName="authorino" containerID="cri-o://05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e" gracePeriod=30
Oct 31 18:27:53.195481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.195391    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" event={"ID":"9525d3e3-170d-42d1-a7cf-89a87b80d0fc","Type":"ContainerStarted","Data":"05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8"}
Oct 31 18:27:53.294720 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.294665    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-58b675c9d7-9zlvm" podStartSLOduration=2.103279601 podStartE2EDuration="3.294650094s" totalImagesPullingTime="1.191370493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:53.29207546 +0000 UTC m=+376.523313762" watchObservedRunningTime="2026-10-31 18:27:53.294650094 +0000 UTC m=+376.525888383"
Oct 31 18:27:53.408906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.408334    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-k7smf" podStartSLOduration=2.826965006 podStartE2EDuration="4.408313852s" totalImagesPullingTime="1.581348846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:53.405609187 +0000 UTC m=+376.636847478" watchObservedRunningTime="2026-10-31 18:27:53.408313852 +0000 UTC m=+376.639552143"
Oct 31 18:27:53.545052 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.544996    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh" podStartSLOduration=2.290580444 podStartE2EDuration="3.544975344s" totalImagesPullingTime="1.2543949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:53.543214326 +0000 UTC m=+376.774452616" watchObservedRunningTime="2026-10-31 18:27:53.544975344 +0000 UTC m=+376.776213637"
Oct 31 18:27:53.548527 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.548496    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b675c9d7-9zlvm"
Oct 31 18:27:53.585487 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.581665    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-k7smf"
Oct 31 18:27:53.592344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.589146    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j5l26"]
Oct 31 18:27:53.592344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.589315    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26" podUID="05aaf52b-dcc8-4a4e-87c1-e211b979dc51" containerName="authorino" containerID="cri-o://ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71" gracePeriod=30
Oct 31 18:27:53.686058 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.686022    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216-kube-api-access-pxfkl\" (UniqueName: \"kubernetes.io/projected/027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216-kube-api-access-pxfkl\") pod \"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216\" (UID: \"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216\") "
Oct 31 18:27:53.686288 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.686088    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a618f2d-f0a8-45aa-9374-2ee554bac60f-kube-api-access-g56pk\" (UniqueName: \"kubernetes.io/projected/5a618f2d-f0a8-45aa-9374-2ee554bac60f-kube-api-access-g56pk\") pod \"5a618f2d-f0a8-45aa-9374-2ee554bac60f\" (UID: \"5a618f2d-f0a8-45aa-9374-2ee554bac60f\") "
Oct 31 18:27:53.688510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.688478    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a618f2d-f0a8-45aa-9374-2ee554bac60f-kube-api-access-g56pk" pod "5a618f2d-f0a8-45aa-9374-2ee554bac60f" (UID: "5a618f2d-f0a8-45aa-9374-2ee554bac60f"). InnerVolumeSpecName "kube-api-access-g56pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:27:53.688826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.688803    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216-kube-api-access-pxfkl" pod "027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216" (UID: "027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216"). InnerVolumeSpecName "kube-api-access-pxfkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:27:53.787410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.787176    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxfkl\" (UniqueName: \"kubernetes.io/projected/027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216-kube-api-access-pxfkl\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:27:53.787410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.787211    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56pk\" (UniqueName: \"kubernetes.io/projected/5a618f2d-f0a8-45aa-9374-2ee554bac60f-kube-api-access-g56pk\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:27:53.862509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.859694    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26"
Oct 31 18:27:53.888032 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.888005    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05aaf52b-dcc8-4a4e-87c1-e211b979dc51-kube-api-access-x7c6x\" (UniqueName: \"kubernetes.io/projected/05aaf52b-dcc8-4a4e-87c1-e211b979dc51-kube-api-access-x7c6x\") pod \"05aaf52b-dcc8-4a4e-87c1-e211b979dc51\" (UID: \"05aaf52b-dcc8-4a4e-87c1-e211b979dc51\") "
Oct 31 18:27:53.890307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.890282    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05aaf52b-dcc8-4a4e-87c1-e211b979dc51-kube-api-access-x7c6x" pod "05aaf52b-dcc8-4a4e-87c1-e211b979dc51" (UID: "05aaf52b-dcc8-4a4e-87c1-e211b979dc51"). InnerVolumeSpecName "kube-api-access-x7c6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:27:53.946766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.946673    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:27:53.947800 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.947767    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rbrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 18:27:53.947934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.947841    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 18:27:53.989024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:53.988995    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7c6x\" (UniqueName: \"kubernetes.io/projected/05aaf52b-dcc8-4a4e-87c1-e211b979dc51-kube-api-access-x7c6x\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:27:54.201811 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.201727    5551 generic.go:356] "Generic (PLEG): container finished" podID="05aaf52b-dcc8-4a4e-87c1-e211b979dc51" containerID="ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71" exitCode=0
Oct 31 18:27:54.201811 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.201771    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26"
Oct 31 18:27:54.202315 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.201809    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26" event={"ID":"05aaf52b-dcc8-4a4e-87c1-e211b979dc51","Type":"ContainerDied","Data":"ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71"}
Oct 31 18:27:54.202315 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.201857    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-j5l26" event={"ID":"05aaf52b-dcc8-4a4e-87c1-e211b979dc51","Type":"ContainerDied","Data":"6c4dc12f8a396aae1b8e651ddbfa9b1ba16a53aeed03cd50ed02518684bfe55f"}
Oct 31 18:27:54.202315 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.201881    5551 scope.go:122] "RemoveContainer" containerID="ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71"
Oct 31 18:27:54.203391 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.203368    5551 generic.go:356] "Generic (PLEG): container finished" podID="027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216" containerID="6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4" exitCode=0
Oct 31 18:27:54.203549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.203396    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58b675c9d7-9zlvm"
Oct 31 18:27:54.203549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.203445    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b675c9d7-9zlvm" event={"ID":"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216","Type":"ContainerDied","Data":"6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4"}
Oct 31 18:27:54.203549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.203512    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58b675c9d7-9zlvm" event={"ID":"027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216","Type":"ContainerDied","Data":"c99bafbf146bd4280ed495e11ef3bd6bf45ba34c4f927b168ca940e7491d0e09"}
Oct 31 18:27:54.205047 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.205009    5551 generic.go:356] "Generic (PLEG): container finished" podID="5a618f2d-f0a8-45aa-9374-2ee554bac60f" containerID="05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e" exitCode=0
Oct 31 18:27:54.205047 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.205036    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-k7smf"
Oct 31 18:27:54.205215 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.205057    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-k7smf" event={"ID":"5a618f2d-f0a8-45aa-9374-2ee554bac60f","Type":"ContainerDied","Data":"05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e"}
Oct 31 18:27:54.205215 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.205096    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-k7smf" event={"ID":"5a618f2d-f0a8-45aa-9374-2ee554bac60f","Type":"ContainerDied","Data":"0d770b4db2042b0dbbea078bd60e8012873b16cade99aa90367d0a4ecd8b3efa"}
Oct 31 18:27:54.212937 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.212863    5551 scope.go:122] "RemoveContainer" containerID="ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71"
Oct 31 18:27:54.213162 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:27:54.213144    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71\": container with ID starting with ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71 not found: ID does not exist" containerID="ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71"
Oct 31 18:27:54.213227 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.213173    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71"} err="failed to get container status \"ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71\": rpc error: code = NotFound desc = could not find container \"ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71\": container with ID starting with ab469b320bafb29c68756eff5e8861c4d8f04da3b7dd71a2a5c5a6e92ee46f71 not found: ID does not exist"
Oct 31 18:27:54.213227 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.213189    5551 scope.go:122] "RemoveContainer" containerID="6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4"
Oct 31 18:27:54.223790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.223772    5551 scope.go:122] "RemoveContainer" containerID="6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4"
Oct 31 18:27:54.224079 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:27:54.224049    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4\": container with ID starting with 6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4 not found: ID does not exist" containerID="6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4"
Oct 31 18:27:54.224116 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.224086    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4"} err="failed to get container status \"6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4\": rpc error: code = NotFound desc = could not find container \"6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4\": container with ID starting with 6e2ef3d69427f7c9d57b3f1ed60065ff903f993d09b92565bb0eaea3619d34b4 not found: ID does not exist"
Oct 31 18:27:54.224116 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.224105    5551 scope.go:122] "RemoveContainer" containerID="05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e"
Oct 31 18:27:54.231895 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.231879    5551 scope.go:122] "RemoveContainer" containerID="05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e"
Oct 31 18:27:54.232183 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:27:54.232165    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e\": container with ID starting with 05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e not found: ID does not exist" containerID="05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e"
Oct 31 18:27:54.232233 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.232190    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e"} err="failed to get container status \"05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e\": rpc error: code = NotFound desc = could not find container \"05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e\": container with ID starting with 05363196a18943777ce8df2f2a7f2591cff419765f509a8ce61d01f1756d004e not found: ID does not exist"
Oct 31 18:27:54.379018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.378975    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j5l26"]
Oct 31 18:27:54.382256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.382230    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j5l26"]
Oct 31 18:27:54.490762 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.490670    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58b675c9d7-9zlvm"]
Oct 31 18:27:54.495277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.495224    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58b675c9d7-9zlvm"]
Oct 31 18:27:54.602705 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.602674    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-k7smf"]
Oct 31 18:27:54.606078 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.606047    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-k7smf"]
Oct 31 18:27:54.908044 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.908005    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216" path="/var/lib/kubelet/pods/027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216/volumes"
Oct 31 18:27:54.908340 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.908327    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05aaf52b-dcc8-4a4e-87c1-e211b979dc51" path="/var/lib/kubelet/pods/05aaf52b-dcc8-4a4e-87c1-e211b979dc51/volumes"
Oct 31 18:27:54.908649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.908637    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a618f2d-f0a8-45aa-9374-2ee554bac60f" path="/var/lib/kubelet/pods/5a618f2d-f0a8-45aa-9374-2ee554bac60f/volumes"
Oct 31 18:27:54.946448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.946413    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rbrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 18:27:54.946621 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:54.946507    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 18:27:55.946117 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:55.946077    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rbrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 18:27:55.946525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:55.946142    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 18:27:56.945802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:56.945764    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rbrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 18:27:56.945982 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:56.945815    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 18:27:57.946212 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:57.946180    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rbrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 18:27:57.946615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:57.946239    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 18:27:57.957572 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:57.957543    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:27:58.949828 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:58.949789    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:27:58.950794 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:27:58.950773    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:28:16.603022 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:28:16.602984    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:28:46.602601 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:28:46.602562    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:29:16.602993 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:29:16.602908    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:29:46.603567 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:29:46.603530    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:30:00.156339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156306    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"]
Oct 31 18:30:00.156837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156743    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5a618f2d-f0a8-45aa-9374-2ee554bac60f" containerName="authorino"
Oct 31 18:30:00.156837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156758    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a618f2d-f0a8-45aa-9374-2ee554bac60f" containerName="authorino"
Oct 31 18:30:00.156837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156769    5551 cpu_manager.go:414] "containerMap: removing container" podUID="05aaf52b-dcc8-4a4e-87c1-e211b979dc51" containerName="authorino"
Oct 31 18:30:00.156837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156774    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05aaf52b-dcc8-4a4e-87c1-e211b979dc51" containerName="authorino"
Oct 31 18:30:00.156837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156783    5551 cpu_manager.go:414] "containerMap: removing container" podUID="027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216" containerName="authorino"
Oct 31 18:30:00.156837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156788    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216" containerName="authorino"
Oct 31 18:30:00.157051 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156861    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a618f2d-f0a8-45aa-9374-2ee554bac60f" containerName="authorino"
Oct 31 18:30:00.157051 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156869    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="05aaf52b-dcc8-4a4e-87c1-e211b979dc51" containerName="authorino"
Oct 31 18:30:00.157051 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.156876    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="027d1bb7-6bbf-4eb9-a2b7-d7eecf6b0216" containerName="authorino"
Oct 31 18:30:00.160074 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.160057    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:00.191111 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.191079    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891190-mjtlx\" (UID: \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:00.191253 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.191235    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4gb\" (UniqueName: \"kubernetes.io/projected/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-kube-api-access-8x4gb\") pod \"maas-api-key-cleanup-29891190-mjtlx\" (UID: \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:00.215677 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.215648    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"]
Oct 31 18:30:00.262218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.262188    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 18:30:00.262218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.262190    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 18:30:00.262411 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.262238    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:30:00.291849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.291822    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891190-mjtlx\" (UID: \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:00.291998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.291925    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4gb\" (UniqueName: \"kubernetes.io/projected/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-kube-api-access-8x4gb\") pod \"maas-api-key-cleanup-29891190-mjtlx\" (UID: \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:00.292555 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.292536    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891190-mjtlx\" (UID: \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:00.348496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.348438    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4gb\" (UniqueName: \"kubernetes.io/projected/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-kube-api-access-8x4gb\") pod \"maas-api-key-cleanup-29891190-mjtlx\" (UID: \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:00.474982 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.474907    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:00.847499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.847476    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"]
Oct 31 18:30:00.850178 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:30:00.850150    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21596cf_69de_4b7a_8c90_ae5cdf5f997d.slice/crio-3e21c74aa204b6aaf828865d969bb6a8d93f07269f7e8908980ec624ffc8aea8 WatchSource:0}: Error finding container 3e21c74aa204b6aaf828865d969bb6a8d93f07269f7e8908980ec624ffc8aea8: Status 404 returned error can't find the container with id 3e21c74aa204b6aaf828865d969bb6a8d93f07269f7e8908980ec624ffc8aea8
Oct 31 18:30:00.890383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.890347    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"]
Oct 31 18:30:00.894811 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.894792    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:00.899974 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.899943    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx" event={"ID":"f21596cf-69de-4b7a-8c90-ae5cdf5f997d","Type":"ContainerStarted","Data":"3e21c74aa204b6aaf828865d969bb6a8d93f07269f7e8908980ec624ffc8aea8"}
Oct 31 18:30:00.949002 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.948971    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 18:30:00.949210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.949027    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 18:30:00.949210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.949193    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 18:30:00.949321 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.949286    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v7tpx\""
Oct 31 18:30:00.951618 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.951596    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"]
Oct 31 18:30:00.999976 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.999948    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.000101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:00.999996    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.000101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.000064    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhhp\" (UniqueName: \"kubernetes.io/projected/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-kube-api-access-7bhhp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.000213 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.000174    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.000300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.000285    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.000362 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.000349    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.101606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.101538    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.101606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.101603    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.101789 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.101633    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.101789 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.101662    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.101789 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.101715    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.101789 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.101741    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhhp\" (UniqueName: \"kubernetes.io/projected/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-kube-api-access-7bhhp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.102006 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.101965    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.102064 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.102007    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.102113 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.102069    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.103975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.103956    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.104224 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.104205    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.158041 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.158004    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhhp\" (UniqueName: \"kubernetes.io/projected/bd7eec1c-4593-41c1-8eca-cbf856f87ab5-kube-api-access-7bhhp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wj9dz\" (UID: \"bd7eec1c-4593-41c1-8eca-cbf856f87ab5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.209915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.209875    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:01.586677 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.586650    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"]
Oct 31 18:30:01.588683 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:30:01.588655    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7eec1c_4593_41c1_8eca_cbf856f87ab5.slice/crio-436778f8bbe247e7fa40717b9b1f0f0761f59f5965d8b14ff7d672daa4251b68 WatchSource:0}: Error finding container 436778f8bbe247e7fa40717b9b1f0f0761f59f5965d8b14ff7d672daa4251b68: Status 404 returned error can't find the container with id 436778f8bbe247e7fa40717b9b1f0f0761f59f5965d8b14ff7d672daa4251b68
Oct 31 18:30:01.906869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:01.906832    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz" event={"ID":"bd7eec1c-4593-41c1-8eca-cbf856f87ab5","Type":"ContainerStarted","Data":"436778f8bbe247e7fa40717b9b1f0f0761f59f5965d8b14ff7d672daa4251b68"}
Oct 31 18:30:04.931216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:04.931159    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz" event={"ID":"bd7eec1c-4593-41c1-8eca-cbf856f87ab5","Type":"ContainerStarted","Data":"9db06c19512aae376c3f8a7441207282b4d79b3048a4a96b319d1d09e8169306"}
Oct 31 18:30:05.051197 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:05.051145    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz" podStartSLOduration=2.758903033 podStartE2EDuration="5.051130546s" totalImagesPullingTime="2.292227513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:05.048917087 +0000 UTC m=+508.280155405" watchObservedRunningTime="2026-10-31 18:30:05.051130546 +0000 UTC m=+508.282368836"
Oct 31 18:30:05.210735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:05.210661    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:06.943744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:06.943652    5551 generic.go:356] "Generic (PLEG): container finished" podID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerID="08e9e2d31924911ea9d7ae588dd5493635cbb8d8778215c96a39c9774ea0028c" exitCode=60
Oct 31 18:30:06.944212 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:06.943743    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx" event={"ID":"f21596cf-69de-4b7a-8c90-ae5cdf5f997d","Type":"ContainerDied","Data":"08e9e2d31924911ea9d7ae588dd5493635cbb8d8778215c96a39c9774ea0028c"}
Oct 31 18:30:06.944212 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:06.943996    5551 scope.go:122] "RemoveContainer" containerID="08e9e2d31924911ea9d7ae588dd5493635cbb8d8778215c96a39c9774ea0028c"
Oct 31 18:30:07.951477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:07.951379    5551 generic.go:356] "Generic (PLEG): container finished" podID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerID="b3a569de37006ac9a03a4f90cc0c49bb79865d1bd7ad0c6dbf09801e5266f916" exitCode=60
Oct 31 18:30:07.951477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:07.951446    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx" event={"ID":"f21596cf-69de-4b7a-8c90-ae5cdf5f997d","Type":"ContainerDied","Data":"b3a569de37006ac9a03a4f90cc0c49bb79865d1bd7ad0c6dbf09801e5266f916"}
Oct 31 18:30:07.951957 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:07.951510    5551 scope.go:122] "RemoveContainer" containerID="08e9e2d31924911ea9d7ae588dd5493635cbb8d8778215c96a39c9774ea0028c"
Oct 31 18:30:07.951957 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:07.951692    5551 scope.go:122] "RemoveContainer" containerID="b3a569de37006ac9a03a4f90cc0c49bb79865d1bd7ad0c6dbf09801e5266f916"
Oct 31 18:30:07.951957 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:30:07.951929    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891190-mjtlx_odh-ai-gateway-infra(f21596cf-69de-4b7a-8c90-ae5cdf5f997d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d"
Oct 31 18:30:08.532197 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.532160    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"]
Oct 31 18:30:08.537230 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.537205    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.640711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.640682    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 18:30:08.643975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.643950    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"]
Oct 31 18:30:08.677117 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.677081    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.677284 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.677227    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.677358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.677311    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtcbd\" (UniqueName: \"kubernetes.io/projected/8c0be12c-0277-48af-881a-0e0da6c70c2f-kube-api-access-dtcbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.677420 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.677395    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.677510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.677437    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0be12c-0277-48af-881a-0e0da6c70c2f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.677598 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.677574    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.778765 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.778721    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.778953 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.778827    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.778953 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.778897    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtcbd\" (UniqueName: \"kubernetes.io/projected/8c0be12c-0277-48af-881a-0e0da6c70c2f-kube-api-access-dtcbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.778953 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.778939    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.779141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.778974    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0be12c-0277-48af-881a-0e0da6c70c2f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.779141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.779015    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.779248 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.779140    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.779248 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.779148    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.779409 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.779386    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.781528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.781505    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c0be12c-0277-48af-881a-0e0da6c70c2f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.782270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.782218    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0be12c-0277-48af-881a-0e0da6c70c2f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.839941 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.839895    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtcbd\" (UniqueName: \"kubernetes.io/projected/8c0be12c-0277-48af-881a-0e0da6c70c2f-kube-api-access-dtcbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr\" (UID: \"8c0be12c-0277-48af-881a-0e0da6c70c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.853524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.853474    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:08.960714 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:08.960019    5551 scope.go:122] "RemoveContainer" containerID="b3a569de37006ac9a03a4f90cc0c49bb79865d1bd7ad0c6dbf09801e5266f916"
Oct 31 18:30:08.960714 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:30:08.960331    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891190-mjtlx_odh-ai-gateway-infra(f21596cf-69de-4b7a-8c90-ae5cdf5f997d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d"
Oct 31 18:30:09.237066 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:09.237028    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"]
Oct 31 18:30:09.240727 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:30:09.240693    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0be12c_0277_48af_881a_0e0da6c70c2f.slice/crio-74605476f83620598bbe5fcb8c49d6c87eb373b9d3ed81cdb66514bfed22c277 WatchSource:0}: Error finding container 74605476f83620598bbe5fcb8c49d6c87eb373b9d3ed81cdb66514bfed22c277: Status 404 returned error can't find the container with id 74605476f83620598bbe5fcb8c49d6c87eb373b9d3ed81cdb66514bfed22c277
Oct 31 18:30:09.968063 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:09.967962    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr" event={"ID":"8c0be12c-0277-48af-881a-0e0da6c70c2f","Type":"ContainerStarted","Data":"0df3578dd70478eb53cdc7b4e1aae559c649a4b9c06b874f5a632c9533c65694"}
Oct 31 18:30:09.968063 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:09.968009    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr" event={"ID":"8c0be12c-0277-48af-881a-0e0da6c70c2f","Type":"ContainerStarted","Data":"74605476f83620598bbe5fcb8c49d6c87eb373b9d3ed81cdb66514bfed22c277"}
Oct 31 18:30:10.083787 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:10.083729    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr" podStartSLOduration=1.484501868 podStartE2EDuration="2.083713546s" totalImagesPullingTime="599.211678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:10.082667159 +0000 UTC m=+513.313905450" watchObservedRunningTime="2026-10-31 18:30:10.083713546 +0000 UTC m=+513.314951836"
Oct 31 18:30:10.854277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:10.854241    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:11.210188 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:11.210158    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:11.222598 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:11.222578    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:11.988679 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:11.988654    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wj9dz"
Oct 31 18:30:12.156449 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.156419    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2whxh"]
Oct 31 18:30:12.160237 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.160220    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.214637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.214608    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 18:30:12.215157 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.215130    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-home\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.215251 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.215192    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.215362 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.215344    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.215477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.215377    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.215556 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.215511    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.215676 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.215659    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2pd\" (UniqueName: \"kubernetes.io/projected/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-kube-api-access-tj2pd\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.217833 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.217813    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2whxh"]
Oct 31 18:30:12.317258 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317186    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-home\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.317258 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317229    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.317577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317267    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.317577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317295    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.317577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317320    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.317577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317369    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2pd\" (UniqueName: \"kubernetes.io/projected/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-kube-api-access-tj2pd\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.317780 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317669    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-home\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.317780 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317714    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.317780 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.317739    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.319866 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.319838    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.319963 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.319911    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.378315 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.378288    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2pd\" (UniqueName: \"kubernetes.io/projected/2ed09c00-e8fa-4c44-a99c-aaf2d91a5153-kube-api-access-tj2pd\") pod \"echo-simulator-kserve-64b6f8f96c-2whxh\" (UID: \"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.474052 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.474010    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:12.850605 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.850580    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2whxh"]
Oct 31 18:30:12.852945 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:30:12.852914    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed09c00_e8fa_4c44_a99c_aaf2d91a5153.slice/crio-3a116ef646046fa9d8590e29e9071f9dc3596684b3841ceedbeef7cf6d8cbba4 WatchSource:0}: Error finding container 3a116ef646046fa9d8590e29e9071f9dc3596684b3841ceedbeef7cf6d8cbba4: Status 404 returned error can't find the container with id 3a116ef646046fa9d8590e29e9071f9dc3596684b3841ceedbeef7cf6d8cbba4
Oct 31 18:30:12.984718 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:12.984680    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh" event={"ID":"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153","Type":"ContainerStarted","Data":"3a116ef646046fa9d8590e29e9071f9dc3596684b3841ceedbeef7cf6d8cbba4"}
Oct 31 18:30:16.007256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:16.007217    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh" event={"ID":"2ed09c00-e8fa-4c44-a99c-aaf2d91a5153","Type":"ContainerStarted","Data":"3cd7675baefd40f691180b36451eb04a1fb99b37f72ed5c05b77d28b991d1e55"}
Oct 31 18:30:16.138504 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:16.130131    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh" podStartSLOduration=1.492126416 podStartE2EDuration="4.130113067s" totalImagesPullingTime="2.637986651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:16.128700592 +0000 UTC m=+519.359938886" watchObservedRunningTime="2026-10-31 18:30:16.130113067 +0000 UTC m=+519.361351359"
Oct 31 18:30:16.474963 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:16.474919    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:16.606071 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:30:16.606034    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:30:18.854586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:18.854554    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:18.867708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:18.867680    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:19.040934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.040904    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr"
Oct 31 18:30:19.655794 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.655757    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"]
Oct 31 18:30:19.659849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.659827    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.714150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.714118    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 18:30:19.765873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.765839    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"]
Oct 31 18:30:19.797190 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.797155    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.797382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.797218    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.797382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.797335    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nm4d\" (UniqueName: \"kubernetes.io/projected/35c185c9-da97-4e98-a6e3-36664eba8faf-kube-api-access-4nm4d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.797549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.797501    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c185c9-da97-4e98-a6e3-36664eba8faf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.797609 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.797555    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.797679 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.797661    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.898290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898252    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.898719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898303    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.898719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898334    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nm4d\" (UniqueName: \"kubernetes.io/projected/35c185c9-da97-4e98-a6e3-36664eba8faf-kube-api-access-4nm4d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.898719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898423    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c185c9-da97-4e98-a6e3-36664eba8faf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.898719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898487    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.898719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898627    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.899001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898792    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.899001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898834    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.899001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.898977    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.900641 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.900624    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c185c9-da97-4e98-a6e3-36664eba8faf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.900959 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.900944    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c185c9-da97-4e98-a6e3-36664eba8faf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.955829 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.955775    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nm4d\" (UniqueName: \"kubernetes.io/projected/35c185c9-da97-4e98-a6e3-36664eba8faf-kube-api-access-4nm4d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86\" (UID: \"35c185c9-da97-4e98-a6e3-36664eba8faf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:19.971750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:19.971728    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:20.356551 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:20.356530    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"]
Oct 31 18:30:20.358898 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:30:20.358874    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35c185c9_da97_4e98_a6e3_36664eba8faf.slice/crio-b1086b65d91104a76c229f18a0c46c3d1dae15f05e910b03ffc6fea66381998f WatchSource:0}: Error finding container b1086b65d91104a76c229f18a0c46c3d1dae15f05e910b03ffc6fea66381998f: Status 404 returned error can't find the container with id b1086b65d91104a76c229f18a0c46c3d1dae15f05e910b03ffc6fea66381998f
Oct 31 18:30:21.048201 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:21.048172    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" event={"ID":"35c185c9-da97-4e98-a6e3-36664eba8faf","Type":"ContainerStarted","Data":"dff4366312f1cdc93b4a0e145ed98a87cf6dfedc4c11bcd1bba6a52fd8c13792"}
Oct 31 18:30:21.048596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:21.048213    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" event={"ID":"35c185c9-da97-4e98-a6e3-36664eba8faf","Type":"ContainerStarted","Data":"b1086b65d91104a76c229f18a0c46c3d1dae15f05e910b03ffc6fea66381998f"}
Oct 31 18:30:21.165584 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:21.165512    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" podStartSLOduration=1.587040676 podStartE2EDuration="2.165492812s" totalImagesPullingTime="578.452136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:21.162618377 +0000 UTC m=+524.393856662" watchObservedRunningTime="2026-10-31 18:30:21.165492812 +0000 UTC m=+524.396731103"
Oct 31 18:30:21.901498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:21.901452    5551 scope.go:122] "RemoveContainer" containerID="b3a569de37006ac9a03a4f90cc0c49bb79865d1bd7ad0c6dbf09801e5266f916"
Oct 31 18:30:21.972061 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:21.972021    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:22.474783 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:22.474710    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:22.487185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:22.487152    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:23.061838 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:23.061804    5551 generic.go:356] "Generic (PLEG): container finished" podID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerID="1145920f551f56cb4a7cb9aea48fb0b3c73aeff7d22dc42506af73cbf5bef76a" exitCode=60
Oct 31 18:30:23.062014 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:23.061869    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx" event={"ID":"f21596cf-69de-4b7a-8c90-ae5cdf5f997d","Type":"ContainerDied","Data":"1145920f551f56cb4a7cb9aea48fb0b3c73aeff7d22dc42506af73cbf5bef76a"}
Oct 31 18:30:23.062014 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:23.061951    5551 scope.go:122] "RemoveContainer" containerID="b3a569de37006ac9a03a4f90cc0c49bb79865d1bd7ad0c6dbf09801e5266f916"
Oct 31 18:30:23.062939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:23.062918    5551 scope.go:122] "RemoveContainer" containerID="1145920f551f56cb4a7cb9aea48fb0b3c73aeff7d22dc42506af73cbf5bef76a"
Oct 31 18:30:23.063200 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:30:23.063148    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891190-mjtlx_odh-ai-gateway-infra(f21596cf-69de-4b7a-8c90-ae5cdf5f997d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d"
Oct 31 18:30:23.078481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:23.075179    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2whxh"
Oct 31 18:30:24.187105 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:24.187062    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"]
Oct 31 18:30:24.336980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:24.336947    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:24.454534 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:24.454423    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-kube-api-access-8x4gb\" (UniqueName: \"kubernetes.io/projected/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-kube-api-access-8x4gb\") pod \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\" (UID: \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\") "
Oct 31 18:30:24.454709 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:24.454651    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-maas-api-service-ca\") pod \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\" (UID: \"f21596cf-69de-4b7a-8c90-ae5cdf5f997d\") "
Oct 31 18:30:24.455277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:24.455245    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-maas-api-service-ca" pod "f21596cf-69de-4b7a-8c90-ae5cdf5f997d" (UID: "f21596cf-69de-4b7a-8c90-ae5cdf5f997d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:30:24.465049 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:24.465007    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-kube-api-access-8x4gb" pod "f21596cf-69de-4b7a-8c90-ae5cdf5f997d" (UID: "f21596cf-69de-4b7a-8c90-ae5cdf5f997d"). InnerVolumeSpecName "kube-api-access-8x4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:30:24.555825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:24.555796    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x4gb\" (UniqueName: \"kubernetes.io/projected/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-kube-api-access-8x4gb\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:30:24.555825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:24.555820    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21596cf-69de-4b7a-8c90-ae5cdf5f997d-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:30:25.087481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:25.085987    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx" event={"ID":"f21596cf-69de-4b7a-8c90-ae5cdf5f997d","Type":"ContainerDied","Data":"3e21c74aa204b6aaf828865d969bb6a8d93f07269f7e8908980ec624ffc8aea8"}
Oct 31 18:30:25.087481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:25.086066    5551 scope.go:122] "RemoveContainer" containerID="1145920f551f56cb4a7cb9aea48fb0b3c73aeff7d22dc42506af73cbf5bef76a"
Oct 31 18:30:25.087481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:25.086225    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"
Oct 31 18:30:25.254240 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:25.254145    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"]
Oct 31 18:30:25.259109 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:25.259080    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891190-mjtlx"]
Oct 31 18:30:26.907877 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:26.907839    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" path="/var/lib/kubelet/pods/f21596cf-69de-4b7a-8c90-ae5cdf5f997d/volumes"
Oct 31 18:30:29.972336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:29.972299    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:29.984913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:29.984888    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:30.125321 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:30.125295    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:30:45.958413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.958315    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d67dfb554-c2w6p"]
Oct 31 18:30:45.959018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.958828    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:30:45.959018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.958843    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:30:45.959018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.958883    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:30:45.959018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.958890    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:30:45.959018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.958962    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:30:45.959018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.958972    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:30:45.959018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.958980    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:30:45.961098 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:45.961081    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d67dfb554-c2w6p"
Oct 31 18:30:46.017983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.017946    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d67dfb554-c2w6p"]
Oct 31 18:30:46.061802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.061778    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjx7\" (UniqueName: \"kubernetes.io/projected/e22ad853-ddd2-4669-8df9-3c3f67f659f2-kube-api-access-ggjx7\") pod \"authorino-6d67dfb554-c2w6p\" (UID: \"e22ad853-ddd2-4669-8df9-3c3f67f659f2\") " pod="kuadrant-system/authorino-6d67dfb554-c2w6p"
Oct 31 18:30:46.061954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.061915    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e22ad853-ddd2-4669-8df9-3c3f67f659f2-tls-cert\") pod \"authorino-6d67dfb554-c2w6p\" (UID: \"e22ad853-ddd2-4669-8df9-3c3f67f659f2\") " pod="kuadrant-system/authorino-6d67dfb554-c2w6p"
Oct 31 18:30:46.163042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.163010    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjx7\" (UniqueName: \"kubernetes.io/projected/e22ad853-ddd2-4669-8df9-3c3f67f659f2-kube-api-access-ggjx7\") pod \"authorino-6d67dfb554-c2w6p\" (UID: \"e22ad853-ddd2-4669-8df9-3c3f67f659f2\") " pod="kuadrant-system/authorino-6d67dfb554-c2w6p"
Oct 31 18:30:46.163202 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.163144    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e22ad853-ddd2-4669-8df9-3c3f67f659f2-tls-cert\") pod \"authorino-6d67dfb554-c2w6p\" (UID: \"e22ad853-ddd2-4669-8df9-3c3f67f659f2\") " pod="kuadrant-system/authorino-6d67dfb554-c2w6p"
Oct 31 18:30:46.165856 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.165834    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e22ad853-ddd2-4669-8df9-3c3f67f659f2-tls-cert\") pod \"authorino-6d67dfb554-c2w6p\" (UID: \"e22ad853-ddd2-4669-8df9-3c3f67f659f2\") " pod="kuadrant-system/authorino-6d67dfb554-c2w6p"
Oct 31 18:30:46.219471 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.219396    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjx7\" (UniqueName: \"kubernetes.io/projected/e22ad853-ddd2-4669-8df9-3c3f67f659f2-kube-api-access-ggjx7\") pod \"authorino-6d67dfb554-c2w6p\" (UID: \"e22ad853-ddd2-4669-8df9-3c3f67f659f2\") " pod="kuadrant-system/authorino-6d67dfb554-c2w6p"
Oct 31 18:30:46.274175 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.274149    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d67dfb554-c2w6p"
Oct 31 18:30:46.604180 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:30:46.604097    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:30:46.646805 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:46.646783    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d67dfb554-c2w6p"]
Oct 31 18:30:46.649439 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:30:46.649407    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode22ad853_ddd2_4669_8df9_3c3f67f659f2.slice/crio-237c19d9128933c5690b36bef869adf861c99ed32b14d3b6048bd4d4a1d356b1 WatchSource:0}: Error finding container 237c19d9128933c5690b36bef869adf861c99ed32b14d3b6048bd4d4a1d356b1: Status 404 returned error can't find the container with id 237c19d9128933c5690b36bef869adf861c99ed32b14d3b6048bd4d4a1d356b1
Oct 31 18:30:47.206989 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:47.206947    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d67dfb554-c2w6p" event={"ID":"e22ad853-ddd2-4669-8df9-3c3f67f659f2","Type":"ContainerStarted","Data":"237c19d9128933c5690b36bef869adf861c99ed32b14d3b6048bd4d4a1d356b1"}
Oct 31 18:30:49.229987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.229947    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d67dfb554-c2w6p" event={"ID":"e22ad853-ddd2-4669-8df9-3c3f67f659f2","Type":"ContainerStarted","Data":"397dc48edd7efb182ce5e1e9d69c5586d6c4822c968cf5e60906d3f6f2651ed7"}
Oct 31 18:30:49.349331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.347345    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d67dfb554-c2w6p" podStartSLOduration=2.747181032 podStartE2EDuration="4.347325611s" totalImagesPullingTime="1.600144579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:49.347305229 +0000 UTC m=+552.578543520" watchObservedRunningTime="2026-10-31 18:30:49.347325611 +0000 UTC m=+552.578563901"
Oct 31 18:30:49.399494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.398387    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dc886c4c7-ppfbh"]
Oct 31 18:30:49.399494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.398720    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh" podUID="76ff8676-9921-458f-9081-1ea97cf63a31" containerName="authorino" containerID="cri-o://3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e" gracePeriod=30
Oct 31 18:30:49.674578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.674553    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:30:49.807245 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.807168    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ff8676-9921-458f-9081-1ea97cf63a31-kube-api-access-dl9zg\" (UniqueName: \"kubernetes.io/projected/76ff8676-9921-458f-9081-1ea97cf63a31-kube-api-access-dl9zg\") pod \"76ff8676-9921-458f-9081-1ea97cf63a31\" (UID: \"76ff8676-9921-458f-9081-1ea97cf63a31\") "
Oct 31 18:30:49.807245 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.807223    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ff8676-9921-458f-9081-1ea97cf63a31-tls-cert\" (UniqueName: \"kubernetes.io/secret/76ff8676-9921-458f-9081-1ea97cf63a31-tls-cert\") pod \"76ff8676-9921-458f-9081-1ea97cf63a31\" (UID: \"76ff8676-9921-458f-9081-1ea97cf63a31\") "
Oct 31 18:30:49.809444 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.809407    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ff8676-9921-458f-9081-1ea97cf63a31-kube-api-access-dl9zg" pod "76ff8676-9921-458f-9081-1ea97cf63a31" (UID: "76ff8676-9921-458f-9081-1ea97cf63a31"). InnerVolumeSpecName "kube-api-access-dl9zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:30:49.822300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.822261    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ff8676-9921-458f-9081-1ea97cf63a31-tls-cert" pod "76ff8676-9921-458f-9081-1ea97cf63a31" (UID: "76ff8676-9921-458f-9081-1ea97cf63a31"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:30:49.908708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.908662    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl9zg\" (UniqueName: \"kubernetes.io/projected/76ff8676-9921-458f-9081-1ea97cf63a31-kube-api-access-dl9zg\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:30:49.908708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:49.908710    5551 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/76ff8676-9921-458f-9081-1ea97cf63a31-tls-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:30:50.237574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.237544    5551 generic.go:356] "Generic (PLEG): container finished" podID="76ff8676-9921-458f-9081-1ea97cf63a31" containerID="3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e" exitCode=0
Oct 31 18:30:50.237996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.237624    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh"
Oct 31 18:30:50.237996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.237633    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh" event={"ID":"76ff8676-9921-458f-9081-1ea97cf63a31","Type":"ContainerDied","Data":"3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e"}
Oct 31 18:30:50.237996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.237677    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc886c4c7-ppfbh" event={"ID":"76ff8676-9921-458f-9081-1ea97cf63a31","Type":"ContainerDied","Data":"1b5dec908f6cbf6dc56dbb6da2d7fc3e52bf262f32409e0b9162a18e7497b5e7"}
Oct 31 18:30:50.237996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.237693    5551 scope.go:122] "RemoveContainer" containerID="3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e"
Oct 31 18:30:50.248891 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.248875    5551 scope.go:122] "RemoveContainer" containerID="3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e"
Oct 31 18:30:50.249169 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:30:50.249153    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e\": container with ID starting with 3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e not found: ID does not exist" containerID="3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e"
Oct 31 18:30:50.249218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.249181    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e"} err="failed to get container status \"3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e\": rpc error: code = NotFound desc = could not find container \"3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e\": container with ID starting with 3b6d91b14ac10bc012ae1cc61c5f5c72f90c7dec401a9d1cf1c367dbbabb973e not found: ID does not exist"
Oct 31 18:30:50.400788 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.400754    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dc886c4c7-ppfbh"]
Oct 31 18:30:50.449475 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.449437    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7dc886c4c7-ppfbh"]
Oct 31 18:30:50.908578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:30:50.908532    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ff8676-9921-458f-9081-1ea97cf63a31" path="/var/lib/kubelet/pods/76ff8676-9921-458f-9081-1ea97cf63a31/volumes"
Oct 31 18:31:02.572321 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.572286    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"]
Oct 31 18:31:02.572867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.572731    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:31:02.572867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.572743    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21596cf-69de-4b7a-8c90-ae5cdf5f997d" containerName="cleanup"
Oct 31 18:31:02.572867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.572773    5551 cpu_manager.go:414] "containerMap: removing container" podUID="76ff8676-9921-458f-9081-1ea97cf63a31" containerName="authorino"
Oct 31 18:31:02.572867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.572778    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ff8676-9921-458f-9081-1ea97cf63a31" containerName="authorino"
Oct 31 18:31:02.572867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.572847    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ff8676-9921-458f-9081-1ea97cf63a31" containerName="authorino"
Oct 31 18:31:02.577378 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.577358    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.681579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.681545    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"]
Oct 31 18:31:02.739875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.739846    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff2b758-259f-476d-85f6-1785a52e8b67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.739875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.739883    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.740089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.739974    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.740089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.740012    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.740089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.740050    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.740260 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.740182    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.740260 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.740220    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff2b758-259f-476d-85f6-1785a52e8b67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.740365 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.740327    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdnb\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-kube-api-access-fpdnb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.740421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.740365    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841270    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff2b758-259f-476d-85f6-1785a52e8b67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841308    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841598 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841356    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841598 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841387    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841598 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841541    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841605    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841638    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff2b758-259f-476d-85f6-1785a52e8b67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841697    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdnb\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-kube-api-access-fpdnb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841732    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841788    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841814    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.841961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.841879    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.842116 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.842094    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff2b758-259f-476d-85f6-1785a52e8b67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.842191 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.842166    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.844019 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.843997    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.844289 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.844267    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff2b758-259f-476d-85f6-1785a52e8b67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.898811 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.898780    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:02.898994 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:02.898849    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdnb\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-kube-api-access-fpdnb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tk5s8\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:03.191230 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:03.191202    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:03.619291 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:03.619268    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"]
Oct 31 18:31:03.621382 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:03.621357    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff2b758_259f_476d_85f6_1785a52e8b67.slice/crio-6b9202119e746bfe6f25359f49c2b4d3afc5152e83b80d15252d57a6ea406b31 WatchSource:0}: Error finding container 6b9202119e746bfe6f25359f49c2b4d3afc5152e83b80d15252d57a6ea406b31: Status 404 returned error can't find the container with id 6b9202119e746bfe6f25359f49c2b4d3afc5152e83b80d15252d57a6ea406b31
Oct 31 18:31:03.623471 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:03.623430    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:03.623525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:03.623507    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:03.623560 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:03.623538    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:04.321338 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:04.321294    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" event={"ID":"dff2b758-259f-476d-85f6-1785a52e8b67","Type":"ContainerStarted","Data":"112aeb7b4cfb85fc3365226f1b291c4b2a3a10e6fe7e68bdb214f17e97ffcd5a"}
Oct 31 18:31:04.321586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:04.321372    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" event={"ID":"dff2b758-259f-476d-85f6-1785a52e8b67","Type":"ContainerStarted","Data":"6b9202119e746bfe6f25359f49c2b4d3afc5152e83b80d15252d57a6ea406b31"}
Oct 31 18:31:04.538011 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:04.537951    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podStartSLOduration=2.537934629 podStartE2EDuration="2.537934629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:04.486933995 +0000 UTC m=+567.718172297" watchObservedRunningTime="2026-10-31 18:31:04.537934629 +0000 UTC m=+567.769172985"
Oct 31 18:31:05.191932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:05.191893    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:05.193570 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:05.193542    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tk5s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 18:31:05.193699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:05.193601    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 18:31:06.192012 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:06.191979    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tk5s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 18:31:06.192421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:06.192028    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 18:31:07.192481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:07.192430    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tk5s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 18:31:07.192868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:07.192513    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 18:31:08.192771 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:08.192724    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tk5s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 18:31:08.193185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:08.192788    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 18:31:09.192570 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:09.192533    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tk5s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 18:31:09.192770 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:09.192591    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 18:31:10.192327 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:10.192287    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tk5s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 18:31:10.192718 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:10.192381    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 18:31:11.864397 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:11.864368    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:11.864916 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:11.864811    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:11.865709 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:11.865692    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:31:16.603783 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:31:16.603750    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:31:17.321725 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.321690    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"]
Oct 31 18:31:17.325474 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.325439    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.377847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.377818    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bbf5caaf-openshift-default-dockercfg-jbsws\""
Oct 31 18:31:17.413745 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.413716    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-certs\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.413940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.413752    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrnp\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-kube-api-access-jmrnp\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.413940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.413771    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-socket\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.413940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.413794    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-credential-socket\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.413940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.413931    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-podinfo\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.414164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.413956    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-data\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.414164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.414003    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c51df8e-18c3-4017-a30e-c4d895fec593-istiod-ca-cert\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.414164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.414069    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-envoy\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.414164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.414102    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-token\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.429867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.429836    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"]
Oct 31 18:31:17.514879 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.514845    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrnp\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-kube-api-access-jmrnp\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.514879 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.514878    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-socket\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.515416 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.514901    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-credential-socket\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.515416 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.514936    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-podinfo\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.516572 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.515055    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-data\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.516790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.515606    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-data\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.516894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.515955    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-credential-socket\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.516894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.516358    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-socket\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.517021 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.516782    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c51df8e-18c3-4017-a30e-c4d895fec593-istiod-ca-cert\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.517076 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.517064    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-envoy\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.517144 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.517123    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-token\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.517220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.517205    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-certs\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.517595 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.517571    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-certs\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.517683 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.517605    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c51df8e-18c3-4017-a30e-c4d895fec593-istiod-ca-cert\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.518543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.518519    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-podinfo\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.519866 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.519844    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-envoy\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.573451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.573395    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrnp\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-kube-api-access-jmrnp\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.573903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.573886    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-token\") pod \"e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") " pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.639983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.639956    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:17.924364 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.924332    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"]
Oct 31 18:31:17.928350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.928328    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:17.982841 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.982810    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1f4b473d-gw-openshift-default-dockercfg-rjm7q\""
Oct 31 18:31:17.986002 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:17.985981    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"]
Oct 31 18:31:18.021992 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.021965    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d073d45-50fa-48a2-b95d-09d666a92991-istiod-ca-cert\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.022117 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.022005    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-istio-token\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.022271 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.022260    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-credential-socket\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.022359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.022294    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-envoy\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.022449 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.022427    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-socket\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.022560 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.022537    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d073d45-50fa-48a2-b95d-09d666a92991-istio-podinfo\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.022624 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.022573    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-data\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.022624 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.022589    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9fl\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-kube-api-access-lm9fl\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.022624 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.022608    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-certs\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.089609 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.086354    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"]
Oct 31 18:31:18.090080 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:18.090052    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c51df8e_18c3_4017_a30e_c4d895fec593.slice/crio-dbc80c335227746391d5f4c70736a64e5640d196b281a84b6d66e78d0d7a7c7c WatchSource:0}: Error finding container dbc80c335227746391d5f4c70736a64e5640d196b281a84b6d66e78d0d7a7c7c: Status 404 returned error can't find the container with id dbc80c335227746391d5f4c70736a64e5640d196b281a84b6d66e78d0d7a7c7c
Oct 31 18:31:18.092844 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.092813    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:18.092906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.092881    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:18.092950 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.092910    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:18.123451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123427    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-socket\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123480    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d073d45-50fa-48a2-b95d-09d666a92991-istio-podinfo\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123676 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123597    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-data\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123676 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123643    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9fl\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-kube-api-access-lm9fl\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123781 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123675    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-certs\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123781 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123720    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d073d45-50fa-48a2-b95d-09d666a92991-istiod-ca-cert\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123781 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123768    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-istio-token\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123819    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-socket\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123852    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-credential-socket\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123890    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-envoy\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.123933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.123901    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-data\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.124122 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.124012    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-certs\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.124476 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.124437    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-credential-socket\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.124553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.124536    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d073d45-50fa-48a2-b95d-09d666a92991-istiod-ca-cert\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.126245 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.126218    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d073d45-50fa-48a2-b95d-09d666a92991-istio-podinfo\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.126480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.126447    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-envoy\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.180656 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.180597    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-istio-token\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.180934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.180916    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9fl\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-kube-api-access-lm9fl\") pod \"e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") " pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.241396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.241361    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:18.399717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.399002    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" event={"ID":"4c51df8e-18c3-4017-a30e-c4d895fec593","Type":"ContainerStarted","Data":"3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126"}
Oct 31 18:31:18.399717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.399080    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" event={"ID":"4c51df8e-18c3-4017-a30e-c4d895fec593","Type":"ContainerStarted","Data":"dbc80c335227746391d5f4c70736a64e5640d196b281a84b6d66e78d0d7a7c7c"}
Oct 31 18:31:18.516858 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.516757    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" podStartSLOduration=1.516742152 podStartE2EDuration="1.516742152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:18.514805021 +0000 UTC m=+581.746043336" watchObservedRunningTime="2026-10-31 18:31:18.516742152 +0000 UTC m=+581.747980442"
Oct 31 18:31:18.640302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.640260    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:18.672060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.672038    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"]
Oct 31 18:31:18.674062 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:18.674035    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d073d45_50fa_48a2_b95d_09d666a92991.slice/crio-1e1b01b10f255a547ae029d7290d802f805736f57896804fb4b8d169f244c6d8 WatchSource:0}: Error finding container 1e1b01b10f255a547ae029d7290d802f805736f57896804fb4b8d169f244c6d8: Status 404 returned error can't find the container with id 1e1b01b10f255a547ae029d7290d802f805736f57896804fb4b8d169f244c6d8
Oct 31 18:31:18.676523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.676494    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:18.676615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.676562    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:18.676615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:18.676591    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:19.406390 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:19.406347    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" event={"ID":"8d073d45-50fa-48a2-b95d-09d666a92991","Type":"ContainerStarted","Data":"b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e"}
Oct 31 18:31:19.406581 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:19.406435    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" event={"ID":"8d073d45-50fa-48a2-b95d-09d666a92991","Type":"ContainerStarted","Data":"1e1b01b10f255a547ae029d7290d802f805736f57896804fb4b8d169f244c6d8"}
Oct 31 18:31:19.527941 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:19.527889    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" podStartSLOduration=2.527875529 podStartE2EDuration="2.527875529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:19.525192752 +0000 UTC m=+582.756431129" watchObservedRunningTime="2026-10-31 18:31:19.527875529 +0000 UTC m=+582.759113818"
Oct 31 18:31:19.645936 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:19.645906    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:20.242450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:20.242417    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:20.247386 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:20.247365    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:20.411620 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:20.411589    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:20.411769 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:20.411647    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:20.412331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:20.412309    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:20.412443 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:20.412372    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:24.221827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.221793    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"]
Oct 31 18:31:24.225752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.225734    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.289822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.289776    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-istio-token\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.289998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.289837    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-certs\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.289998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.289892    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-credential-socket\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.289998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.289930    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-envoy\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.290156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.290017    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xxj\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-kube-api-access-z8xxj\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.290156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.290080    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-socket\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.290156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.290139    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73976601-d3aa-41b4-a77e-eea3774707ac-istiod-ca-cert\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.290293 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.290192    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-data\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.290293 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.290250    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73976601-d3aa-41b4-a77e-eea3774707ac-istio-podinfo\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.295248 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.295221    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"]
Oct 31 18:31:24.330837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.330813    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3c7a8371-openshift-default-dockercfg-dvlkm\""
Oct 31 18:31:24.391701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.391672    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-credential-socket\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.391896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.391713    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-envoy\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.391896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.391761    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xxj\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-kube-api-access-z8xxj\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.391896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.391796    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-socket\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392062 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.391932    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73976601-d3aa-41b4-a77e-eea3774707ac-istiod-ca-cert\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392062 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.391981    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-data\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392062 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.392025    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73976601-d3aa-41b4-a77e-eea3774707ac-istio-podinfo\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.392099    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-istio-token\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.392125    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-credential-socket\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.392150    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-certs\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.392276    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-socket\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.392380    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-data\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.392431    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-certs\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.392739 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.392718    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73976601-d3aa-41b4-a77e-eea3774707ac-istiod-ca-cert\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.394898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.394873    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73976601-d3aa-41b4-a77e-eea3774707ac-istio-podinfo\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.394996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.394970    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-envoy\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.448730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.448706    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-istio-token\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.448832 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.448786    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xxj\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-kube-api-access-z8xxj\") pod \"e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") " pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.539526 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.539412    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:24.965539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.965513    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"]
Oct 31 18:31:24.967060 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:24.967032    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73976601_d3aa_41b4_a77e_eea3774707ac.slice/crio-1ebddf258d1237e21cd4e2682589a0d8e4112b0808e01c29cb617811f97bfd0c WatchSource:0}: Error finding container 1ebddf258d1237e21cd4e2682589a0d8e4112b0808e01c29cb617811f97bfd0c: Status 404 returned error can't find the container with id 1ebddf258d1237e21cd4e2682589a0d8e4112b0808e01c29cb617811f97bfd0c
Oct 31 18:31:24.969273 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.969237    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:24.969373 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.969318    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:24.969436 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:24.969369    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:25.441289 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:25.441257    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" event={"ID":"73976601-d3aa-41b4-a77e-eea3774707ac","Type":"ContainerStarted","Data":"0b0f19814d873e1199f3a488fffeaf60480303de40069864696694fd8fd2cea3"}
Oct 31 18:31:25.441679 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:25.441310    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" event={"ID":"73976601-d3aa-41b4-a77e-eea3774707ac","Type":"ContainerStarted","Data":"1ebddf258d1237e21cd4e2682589a0d8e4112b0808e01c29cb617811f97bfd0c"}
Oct 31 18:31:25.540393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:25.540360    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:25.561377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:25.561321    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" podStartSLOduration=1.561302317 podStartE2EDuration="1.561302317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:25.560861006 +0000 UTC m=+588.792099296" watchObservedRunningTime="2026-10-31 18:31:25.561302317 +0000 UTC m=+588.792540628"
Oct 31 18:31:26.545335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:26.545304    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:27.452923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:27.452890    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:27.454049 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:27.454032    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:31:33.916632 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:33.916590    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"]
Oct 31 18:31:33.920733 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:33.920716    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:33.992829 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:33.980647    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"]
Oct 31 18:31:34.002898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.002787    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26ts\" (UniqueName: \"kubernetes.io/projected/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-kube-api-access-k26ts\") pod \"payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5\" (UID: \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:34.003087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.002971    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5\" (UID: \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:34.025691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.025658    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1f4b473d-dockercfg-7mpwd\""
Oct 31 18:31:34.025691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.025680    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1f4b473d\""
Oct 31 18:31:34.078089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.078053    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"]
Oct 31 18:31:34.081930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.081905    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:34.104612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.104584    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26ts\" (UniqueName: \"kubernetes.io/projected/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-kube-api-access-k26ts\") pod \"payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5\" (UID: \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:34.104761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.104709    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5\" (UID: \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:34.105381 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.105346    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5\" (UID: \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:34.134720 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.134686    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"]
Oct 31 18:31:34.161513 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.161491    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26ts\" (UniqueName: \"kubernetes.io/projected/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-kube-api-access-k26ts\") pod \"payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5\" (UID: \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:34.206042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.205955    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdfcq\" (UniqueName: \"kubernetes.io/projected/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-kube-api-access-fdfcq\") pod \"payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw\" (UID: \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:34.206042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.206014    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw\" (UID: \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:34.239900 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.239873    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:34.307326 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.307285    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdfcq\" (UniqueName: \"kubernetes.io/projected/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-kube-api-access-fdfcq\") pod \"payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw\" (UID: \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:34.307474 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.307369    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw\" (UID: \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:34.308431 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.308403    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw\" (UID: \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:34.372290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.372259    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdfcq\" (UniqueName: \"kubernetes.io/projected/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-kube-api-access-fdfcq\") pod \"payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw\" (UID: \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\") " pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:34.395975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.395943    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:34.614295 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.614273    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"]
Oct 31 18:31:34.616282 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:34.616246    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4edc17_6e21_40fe_bbf2_a2bd5a6e18f8.slice/crio-da3946a41a12b7c820b9070e93ec497384e2bf510bdeaaace6cbeddb2427aba4 WatchSource:0}: Error finding container da3946a41a12b7c820b9070e93ec497384e2bf510bdeaaace6cbeddb2427aba4: Status 404 returned error can't find the container with id da3946a41a12b7c820b9070e93ec497384e2bf510bdeaaace6cbeddb2427aba4
Oct 31 18:31:34.769891 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:34.769870    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"]
Oct 31 18:31:34.771791 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:34.771766    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f13d7b4_5f22_4a9d_bf6e_ae97fc40e310.slice/crio-262e10303191df90c7f58a1e3dbec85f6644e8d8b8dc5706dc8b2af495ce125b WatchSource:0}: Error finding container 262e10303191df90c7f58a1e3dbec85f6644e8d8b8dc5706dc8b2af495ce125b: Status 404 returned error can't find the container with id 262e10303191df90c7f58a1e3dbec85f6644e8d8b8dc5706dc8b2af495ce125b
Oct 31 18:31:35.526376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:35.526326    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5" event={"ID":"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8","Type":"ContainerStarted","Data":"da3946a41a12b7c820b9070e93ec497384e2bf510bdeaaace6cbeddb2427aba4"}
Oct 31 18:31:35.527919 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:35.527893    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw" event={"ID":"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310","Type":"ContainerStarted","Data":"262e10303191df90c7f58a1e3dbec85f6644e8d8b8dc5706dc8b2af495ce125b"}
Oct 31 18:31:36.992489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:36.991907    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:31:36.996116 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:36.995949    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:31:38.553542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:38.553484    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw" event={"ID":"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310","Type":"ContainerStarted","Data":"050913234ea3bc41b4925f9593baa653b14fedb9ffa23667e5978e0f13a36447"}
Oct 31 18:31:38.554032 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:38.553702    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:38.555221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:38.555193    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5" event={"ID":"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8","Type":"ContainerStarted","Data":"3679f7951f9361133d9ca9f926c05d29b075d265e66f95903dab4852088f77a5"}
Oct 31 18:31:38.555333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:38.555324    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:38.666906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:38.666855    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw" podStartSLOduration=1.502014493 podStartE2EDuration="4.666838311s" totalImagesPullingTime="3.164823818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:38.665990769 +0000 UTC m=+601.897229071" watchObservedRunningTime="2026-10-31 18:31:38.666838311 +0000 UTC m=+601.898076602"
Oct 31 18:31:38.781097 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:38.781026    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5" podStartSLOduration=2.560368209 podStartE2EDuration="5.781006038s" totalImagesPullingTime="3.220637829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:38.780829023 +0000 UTC m=+602.012067316" watchObservedRunningTime="2026-10-31 18:31:38.781006038 +0000 UTC m=+602.012244389"
Oct 31 18:31:43.361712 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.361667    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"]
Oct 31 18:31:43.365648 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.365630    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:43.430839 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.426159    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"]
Oct 31 18:31:43.431518 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.431005    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2vh4\" (UniqueName: \"kubernetes.io/projected/2399694e-1948-40ed-bb1c-5220115f72d1-kube-api-access-b2vh4\") pod \"payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc\" (UID: \"2399694e-1948-40ed-bb1c-5220115f72d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:43.431518 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.431059    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399694e-1948-40ed-bb1c-5220115f72d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc\" (UID: \"2399694e-1948-40ed-bb1c-5220115f72d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:43.470837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.470804    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3c7a8371-dockercfg-wxb47\""
Oct 31 18:31:43.474567 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.471137    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3c7a8371\""
Oct 31 18:31:43.508362 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.508335    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"]
Oct 31 18:31:43.512124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.512110    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:43.532011 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.531985    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2vh4\" (UniqueName: \"kubernetes.io/projected/2399694e-1948-40ed-bb1c-5220115f72d1-kube-api-access-b2vh4\") pod \"payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc\" (UID: \"2399694e-1948-40ed-bb1c-5220115f72d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:43.532207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.532170    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399694e-1948-40ed-bb1c-5220115f72d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc\" (UID: \"2399694e-1948-40ed-bb1c-5220115f72d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:43.533163 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.533131    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399694e-1948-40ed-bb1c-5220115f72d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc\" (UID: \"2399694e-1948-40ed-bb1c-5220115f72d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:43.572486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.572445    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"]
Oct 31 18:31:43.588354 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.588331    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2vh4\" (UniqueName: \"kubernetes.io/projected/2399694e-1948-40ed-bb1c-5220115f72d1-kube-api-access-b2vh4\") pod \"payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc\" (UID: \"2399694e-1948-40ed-bb1c-5220115f72d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:43.633533 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.633510    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzv2t\" (UniqueName: \"kubernetes.io/projected/b28190c0-6d04-4edd-95a1-a6646791ce7d-kube-api-access-rzv2t\") pod \"payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl\" (UID: \"b28190c0-6d04-4edd-95a1-a6646791ce7d\") " pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:43.633660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.633640    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28190c0-6d04-4edd-95a1-a6646791ce7d-plugins-config-volume\") pod \"payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl\" (UID: \"b28190c0-6d04-4edd-95a1-a6646791ce7d\") " pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:43.678465 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.678437    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:43.735220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.735183    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28190c0-6d04-4edd-95a1-a6646791ce7d-plugins-config-volume\") pod \"payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl\" (UID: \"b28190c0-6d04-4edd-95a1-a6646791ce7d\") " pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:43.735347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.735318    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzv2t\" (UniqueName: \"kubernetes.io/projected/b28190c0-6d04-4edd-95a1-a6646791ce7d-kube-api-access-rzv2t\") pod \"payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl\" (UID: \"b28190c0-6d04-4edd-95a1-a6646791ce7d\") " pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:43.736143 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.736121    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28190c0-6d04-4edd-95a1-a6646791ce7d-plugins-config-volume\") pod \"payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl\" (UID: \"b28190c0-6d04-4edd-95a1-a6646791ce7d\") " pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:43.793481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.793055    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzv2t\" (UniqueName: \"kubernetes.io/projected/b28190c0-6d04-4edd-95a1-a6646791ce7d-kube-api-access-rzv2t\") pod \"payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl\" (UID: \"b28190c0-6d04-4edd-95a1-a6646791ce7d\") " pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:43.826316 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:43.826288    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:44.055226 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.055196    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"]
Oct 31 18:31:44.056926 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:44.056900    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2399694e_1948_40ed_bb1c_5220115f72d1.slice/crio-887dbc39b4e14f0d50eaa288ed58131ab18f5fd17ece2abe8c07b05258abc43d WatchSource:0}: Error finding container 887dbc39b4e14f0d50eaa288ed58131ab18f5fd17ece2abe8c07b05258abc43d: Status 404 returned error can't find the container with id 887dbc39b4e14f0d50eaa288ed58131ab18f5fd17ece2abe8c07b05258abc43d
Oct 31 18:31:44.212768 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.212741    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"]
Oct 31 18:31:44.214268 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:44.214235    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28190c0_6d04_4edd_95a1_a6646791ce7d.slice/crio-b332c7df7fc55ec8555676819c3c99e5be13e82ef3b460e5f067b93c90b916df WatchSource:0}: Error finding container b332c7df7fc55ec8555676819c3c99e5be13e82ef3b460e5f067b93c90b916df: Status 404 returned error can't find the container with id b332c7df7fc55ec8555676819c3c99e5be13e82ef3b460e5f067b93c90b916df
Oct 31 18:31:44.600186 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.600104    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc" event={"ID":"2399694e-1948-40ed-bb1c-5220115f72d1","Type":"ContainerStarted","Data":"d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264"}
Oct 31 18:31:44.600731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.600707    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc" event={"ID":"2399694e-1948-40ed-bb1c-5220115f72d1","Type":"ContainerStarted","Data":"887dbc39b4e14f0d50eaa288ed58131ab18f5fd17ece2abe8c07b05258abc43d"}
Oct 31 18:31:44.600832 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.600747    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:44.611933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.611906    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl" event={"ID":"b28190c0-6d04-4edd-95a1-a6646791ce7d","Type":"ContainerStarted","Data":"74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1"}
Oct 31 18:31:44.612056 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.611951    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl" event={"ID":"b28190c0-6d04-4edd-95a1-a6646791ce7d","Type":"ContainerStarted","Data":"b332c7df7fc55ec8555676819c3c99e5be13e82ef3b460e5f067b93c90b916df"}
Oct 31 18:31:44.612056 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.612005    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:44.715029 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.714960    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc" podStartSLOduration=1.714944429 podStartE2EDuration="1.714944429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:44.711737368 +0000 UTC m=+607.942975658" watchObservedRunningTime="2026-10-31 18:31:44.714944429 +0000 UTC m=+607.946182722"
Oct 31 18:31:44.846480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:44.834386    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl" podStartSLOduration=1.834366108 podStartE2EDuration="1.834366108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:44.832786986 +0000 UTC m=+608.064025277" watchObservedRunningTime="2026-10-31 18:31:44.834366108 +0000 UTC m=+608.065604399"
Oct 31 18:31:46.322535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.321152    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"]
Oct 31 18:31:46.324914 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.324893    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.443730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.443699    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 18:31:46.443916 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.443732    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 18:31:46.443916 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.443769    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3c7a8371-serving-cert\""
Oct 31 18:31:46.443916 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.443822    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 18:31:46.443916 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.443875    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 18:31:46.447227 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.447201    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"]
Oct 31 18:31:46.522982 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.522942    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-maas-api-tls\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.523232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.523005    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53dc43-7512-4520-ac0c-542ece3c6300-openshift-service-ca\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.523232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.523051    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-metrics-certs\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.523232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.523118    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4btt\" (UniqueName: \"kubernetes.io/projected/0f53dc43-7512-4520-ac0c-542ece3c6300-kube-api-access-v4btt\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.624024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.623923    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4btt\" (UniqueName: \"kubernetes.io/projected/0f53dc43-7512-4520-ac0c-542ece3c6300-kube-api-access-v4btt\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.624181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.624107    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-maas-api-tls\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.624181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.624153    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53dc43-7512-4520-ac0c-542ece3c6300-openshift-service-ca\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.624310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.624193    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-metrics-certs\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.625247 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.625200    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53dc43-7512-4520-ac0c-542ece3c6300-openshift-service-ca\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.628135 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.628094    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-metrics-certs\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.629020 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.628998    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-maas-api-tls\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.640258 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:31:46.640229    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:31:46.689271 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.689195    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4btt\" (UniqueName: \"kubernetes.io/projected/0f53dc43-7512-4520-ac0c-542ece3c6300-kube-api-access-v4btt\") pod \"maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:46.939307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:46.939270    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:47.388970 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:31:47.386994    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f53dc43_7512_4520_ac0c_542ece3c6300.slice/crio-9cec089898f23598f056161dd03e22a7f578df118c0ce65f4c793f43711a47b6 WatchSource:0}: Error finding container 9cec089898f23598f056161dd03e22a7f578df118c0ce65f4c793f43711a47b6: Status 404 returned error can't find the container with id 9cec089898f23598f056161dd03e22a7f578df118c0ce65f4c793f43711a47b6
Oct 31 18:31:47.388970 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:47.387733    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"]
Oct 31 18:31:47.631295 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:47.631261    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc" event={"ID":"0f53dc43-7512-4520-ac0c-542ece3c6300","Type":"ContainerStarted","Data":"9cec089898f23598f056161dd03e22a7f578df118c0ce65f4c793f43711a47b6"}
Oct 31 18:31:49.600523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:49.596566    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:31:49.602369 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:49.601657    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:31:52.674185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:52.674148    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc" event={"ID":"0f53dc43-7512-4520-ac0c-542ece3c6300","Type":"ContainerStarted","Data":"e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8"}
Oct 31 18:31:52.674740 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:52.674220    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:52.789305 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:52.789248    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc" podStartSLOduration=2.373025495 podStartE2EDuration="6.78923129s" totalImagesPullingTime="4.416205795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:52.788266605 +0000 UTC m=+616.019504918" watchObservedRunningTime="2026-10-31 18:31:52.78923129 +0000 UTC m=+616.020469583"
Oct 31 18:31:54.693829 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.693760    5551 generic.go:356] "Generic (PLEG): container finished" podID="8d073d45-50fa-48a2-b95d-09d666a92991" containerID="b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e" exitCode=0
Oct 31 18:31:54.694399 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.693871    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" event={"ID":"8d073d45-50fa-48a2-b95d-09d666a92991","Type":"ContainerDied","Data":"b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e"}
Oct 31 18:31:54.696314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.696288    5551 generic.go:356] "Generic (PLEG): container finished" podID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerID="3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126" exitCode=0
Oct 31 18:31:54.696437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.696347    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" event={"ID":"4c51df8e-18c3-4017-a30e-c4d895fec593","Type":"ContainerDied","Data":"3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126"}
Oct 31 18:31:54.696813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.696793    5551 scope.go:122] "RemoveContainer" containerID="3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126"
Oct 31 18:31:54.697831 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.697593    5551 scope.go:122] "RemoveContainer" containerID="b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e"
Oct 31 18:31:54.698436 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.698405    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:54.698596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.698513    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:54.698596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.698563    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:54.700448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.700403    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:54.700753 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.700729    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:54.700944 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:54.700927    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:31:55.626277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:55.626243    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:31:55.626492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:55.626340    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:31:55.723543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:55.722808    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" event={"ID":"4c51df8e-18c3-4017-a30e-c4d895fec593","Type":"ContainerStarted","Data":"0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc"}
Oct 31 18:31:55.729540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:55.726903    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" event={"ID":"8d073d45-50fa-48a2-b95d-09d666a92991","Type":"ContainerStarted","Data":"d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d"}
Oct 31 18:31:56.242889 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:56.242856    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:31:56.243253 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:56.243229    5551 patch_prober.go:29] interesting pod/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 18:31:56.243411 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:56.243387    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 18:31:56.640954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:56.640929    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:31:56.641508 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:56.641487    5551 patch_prober.go:29] interesting pod/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 18:31:56.641701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:56.641680    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 18:31:57.241814 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:57.241778    5551 patch_prober.go:29] interesting pod/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 18:31:57.242310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:57.241854    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 18:31:57.643107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:57.643053    5551 patch_prober.go:29] interesting pod/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 18:31:57.643287 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:57.643130    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 18:31:58.242439 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:58.242399    5551 patch_prober.go:29] interesting pod/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 18:31:58.242866 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:58.242483    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 18:31:58.640713 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:58.640673    5551 patch_prober.go:29] interesting pod/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 18:31:58.640910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:58.640731    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 18:31:58.704481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:58.703612    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:31:59.249104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:59.249068    5551 patch_prober.go:29] interesting pod/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 18:31:59.249625 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:59.249142    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 18:31:59.641335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:59.641299    5551 patch_prober.go:29] interesting pod/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 18:31:59.641538 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:31:59.641360    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 18:32:00.242597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:00.242557    5551 patch_prober.go:29] interesting pod/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 18:32:00.242781 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:00.242619    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 18:32:00.640972 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:00.640938    5551 patch_prober.go:29] interesting pod/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 18:32:00.641343 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:00.641014    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 18:32:01.246958 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:01.246930    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:32:01.247405 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:01.247376    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:32:01.248334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:01.248313    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:32:01.644925 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:01.644897    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:32:01.645289 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:01.645202    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:32:01.645978 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:01.645952    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:32:02.152355 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.152309    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"]
Oct 31 18:32:02.160408 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.160371    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.207510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.207474    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"]
Oct 31 18:32:02.212733 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.212704    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:32:02.212907 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.212782    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:32:02.228968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.228932    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfl7\" (UniqueName: \"kubernetes.io/projected/b8e499f0-111d-443b-ac68-da3056572047-kube-api-access-8qfl7\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.229169 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.228994    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e499f0-111d-443b-ac68-da3056572047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.229169 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.229083    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e499f0-111d-443b-ac68-da3056572047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.329939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.329894    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfl7\" (UniqueName: \"kubernetes.io/projected/b8e499f0-111d-443b-ac68-da3056572047-kube-api-access-8qfl7\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.330134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.329962    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e499f0-111d-443b-ac68-da3056572047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.330134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.330015    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e499f0-111d-443b-ac68-da3056572047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.330885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.330861    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e499f0-111d-443b-ac68-da3056572047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.332709 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.332678    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e499f0-111d-443b-ac68-da3056572047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.387847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.387809    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfl7\" (UniqueName: \"kubernetes.io/projected/b8e499f0-111d-443b-ac68-da3056572047-kube-api-access-8qfl7\") pod \"maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.473813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.473724    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:02.852435 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:02.852407    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"]
Oct 31 18:32:02.855353 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:32:02.855323    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e499f0_111d_443b_ac68_da3056572047.slice/crio-ca6aa966d1fae0f7df42a77ac821e4148bc3c456eef90a28878f63cbf9a1194b WatchSource:0}: Error finding container ca6aa966d1fae0f7df42a77ac821e4148bc3c456eef90a28878f63cbf9a1194b: Status 404 returned error can't find the container with id ca6aa966d1fae0f7df42a77ac821e4148bc3c456eef90a28878f63cbf9a1194b
Oct 31 18:32:03.840187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:03.840157    5551 generic.go:356] "Generic (PLEG): container finished" podID="b8e499f0-111d-443b-ac68-da3056572047" containerID="e490350868a81cb9eda6438a3b8482bd55baeaadaf13052f2fad8643c74bd685" exitCode=0
Oct 31 18:32:03.840328 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:03.840201    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45" event={"ID":"b8e499f0-111d-443b-ac68-da3056572047","Type":"ContainerDied","Data":"e490350868a81cb9eda6438a3b8482bd55baeaadaf13052f2fad8643c74bd685"}
Oct 31 18:32:03.840328 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:03.840243    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45" event={"ID":"b8e499f0-111d-443b-ac68-da3056572047","Type":"ContainerStarted","Data":"ca6aa966d1fae0f7df42a77ac821e4148bc3c456eef90a28878f63cbf9a1194b"}
Oct 31 18:32:04.848096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:04.847999    5551 generic.go:356] "Generic (PLEG): container finished" podID="73976601-d3aa-41b4-a77e-eea3774707ac" containerID="0b0f19814d873e1199f3a488fffeaf60480303de40069864696694fd8fd2cea3" exitCode=0
Oct 31 18:32:04.848096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:04.848077    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" event={"ID":"73976601-d3aa-41b4-a77e-eea3774707ac","Type":"ContainerDied","Data":"0b0f19814d873e1199f3a488fffeaf60480303de40069864696694fd8fd2cea3"}
Oct 31 18:32:04.848895 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:04.848602    5551 scope.go:122] "RemoveContainer" containerID="0b0f19814d873e1199f3a488fffeaf60480303de40069864696694fd8fd2cea3"
Oct 31 18:32:04.849979 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:04.849951    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:04.850070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:04.850023    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:04.850070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:04.850063    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:05.028838 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.028811    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:05.059889 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.059863    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8e499f0-111d-443b-ac68-da3056572047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e499f0-111d-443b-ac68-da3056572047-maas-api-xks-ca\") pod \"b8e499f0-111d-443b-ac68-da3056572047\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") "
Oct 31 18:32:05.060070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.060019    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e499f0-111d-443b-ac68-da3056572047-kube-api-access-8qfl7\" (UniqueName: \"kubernetes.io/projected/b8e499f0-111d-443b-ac68-da3056572047-kube-api-access-8qfl7\") pod \"b8e499f0-111d-443b-ac68-da3056572047\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") "
Oct 31 18:32:05.060135 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.060124    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e499f0-111d-443b-ac68-da3056572047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e499f0-111d-443b-ac68-da3056572047-maas-api-openshift-ca\") pod \"b8e499f0-111d-443b-ac68-da3056572047\" (UID: \"b8e499f0-111d-443b-ac68-da3056572047\") "
Oct 31 18:32:05.060499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.060447    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e499f0-111d-443b-ac68-da3056572047-maas-api-openshift-ca" pod "b8e499f0-111d-443b-ac68-da3056572047" (UID: "b8e499f0-111d-443b-ac68-da3056572047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:05.062498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.062429    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e499f0-111d-443b-ac68-da3056572047-kube-api-access-8qfl7" pod "b8e499f0-111d-443b-ac68-da3056572047" (UID: "b8e499f0-111d-443b-ac68-da3056572047"). InnerVolumeSpecName "kube-api-access-8qfl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:05.062607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.062526    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e499f0-111d-443b-ac68-da3056572047-maas-api-xks-ca" pod "b8e499f0-111d-443b-ac68-da3056572047" (UID: "b8e499f0-111d-443b-ac68-da3056572047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:05.162249 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.162209    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qfl7\" (UniqueName: \"kubernetes.io/projected/b8e499f0-111d-443b-ac68-da3056572047-kube-api-access-8qfl7\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:05.162441 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.162250    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e499f0-111d-443b-ac68-da3056572047-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:05.162441 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.162284    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e499f0-111d-443b-ac68-da3056572047-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:05.866500 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.866436    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" event={"ID":"73976601-d3aa-41b4-a77e-eea3774707ac","Type":"ContainerStarted","Data":"108ddb898917e4254d5cf2ef40fc7a14f9db8b0ce6911d85fb0d23acc6f7beab"}
Oct 31 18:32:05.880479 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.873625    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"
Oct 31 18:32:05.899307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.899063    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45" event={"ID":"b8e499f0-111d-443b-ac68-da3056572047","Type":"ContainerDied","Data":"ca6aa966d1fae0f7df42a77ac821e4148bc3c456eef90a28878f63cbf9a1194b"}
Oct 31 18:32:05.899307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:05.899155    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca6aa966d1fae0f7df42a77ac821e4148bc3c456eef90a28878f63cbf9a1194b"
Oct 31 18:32:06.540314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:06.540276    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:32:06.541054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:06.541026    5551 patch_prober.go:29] interesting pod/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 18:32:06.541186 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:06.541076    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 18:32:07.540278 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:07.540238    5551 patch_prober.go:29] interesting pod/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 18:32:07.540691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:07.540294    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 18:32:08.540046 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:08.540011    5551 patch_prober.go:29] interesting pod/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 18:32:08.540221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:08.540067    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 18:32:09.540520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:09.540482    5551 patch_prober.go:29] interesting pod/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 18:32:09.541018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:09.540554    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 18:32:10.540234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:10.540201    5551 patch_prober.go:29] interesting pod/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 18:32:10.540440 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:10.540260    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 18:32:11.544149 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:11.544118    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:32:11.544562 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:11.544534    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:32:11.545255 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:11.545238    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:32:12.806554 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.806451    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"]
Oct 31 18:32:12.807067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.807052    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b8e499f0-111d-443b-ac68-da3056572047" containerName="revoke-keys"
Oct 31 18:32:12.807106 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.807069    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e499f0-111d-443b-ac68-da3056572047" containerName="revoke-keys"
Oct 31 18:32:12.807162 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.807153    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e499f0-111d-443b-ac68-da3056572047" containerName="revoke-keys"
Oct 31 18:32:12.809734 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.809715    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:12.904715 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.904687    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mbx\" (UniqueName: \"kubernetes.io/projected/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-kube-api-access-s6mbx\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:12.904913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.904768    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:12.904913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.904855    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:12.912907 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.912858    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:32:12.913057 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.912993    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:32:12.915619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:12.915599    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"]
Oct 31 18:32:13.006512 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.006442    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mbx\" (UniqueName: \"kubernetes.io/projected/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-kube-api-access-s6mbx\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:13.006880 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.006851    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:13.006993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.006899    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:13.007471 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.007443    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:13.009379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.009363    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:13.063619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.063545    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mbx\" (UniqueName: \"kubernetes.io/projected/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-kube-api-access-s6mbx\") pod \"maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:13.122197 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.122173    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:13.499330 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.499298    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"]
Oct 31 18:32:13.500551 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:32:13.500525    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105763cd_fdcc_434b_a946_dfcfbdcc5e2e.slice/crio-f89738fc13f1d6174b6b3659c0bac10a9efe4786d30704f04bb1d6025422ffc7 WatchSource:0}: Error finding container f89738fc13f1d6174b6b3659c0bac10a9efe4786d30704f04bb1d6025422ffc7: Status 404 returned error can't find the container with id f89738fc13f1d6174b6b3659c0bac10a9efe4786d30704f04bb1d6025422ffc7
Oct 31 18:32:13.966695 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.966656    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs" event={"ID":"105763cd-fdcc-434b-a946-dfcfbdcc5e2e","Type":"ContainerStarted","Data":"3f447a0ad1b7a01a481489eb63abafe25f9a0fc9e8b9db112e8468c47d42f5b5"}
Oct 31 18:32:13.967062 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:13.966716    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs" event={"ID":"105763cd-fdcc-434b-a946-dfcfbdcc5e2e","Type":"ContainerStarted","Data":"f89738fc13f1d6174b6b3659c0bac10a9efe4786d30704f04bb1d6025422ffc7"}
Oct 31 18:32:14.080089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:14.079996    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs" podStartSLOduration=2.079982842 podStartE2EDuration="2.079982842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:14.078516641 +0000 UTC m=+637.309754932" watchObservedRunningTime="2026-10-31 18:32:14.079982842 +0000 UTC m=+637.311221145"
Oct 31 18:32:14.980884 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:14.980844    5551 generic.go:356] "Generic (PLEG): container finished" podID="105763cd-fdcc-434b-a946-dfcfbdcc5e2e" containerID="3f447a0ad1b7a01a481489eb63abafe25f9a0fc9e8b9db112e8468c47d42f5b5" exitCode=0
Oct 31 18:32:14.981320 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:14.980893    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs" event={"ID":"105763cd-fdcc-434b-a946-dfcfbdcc5e2e","Type":"ContainerDied","Data":"3f447a0ad1b7a01a481489eb63abafe25f9a0fc9e8b9db112e8468c47d42f5b5"}
Oct 31 18:32:16.127193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.127161    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:16.242113 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.242029    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-kube-api-access-s6mbx\" (UniqueName: \"kubernetes.io/projected/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-kube-api-access-s6mbx\") pod \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") "
Oct 31 18:32:16.242270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.242213    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-xks-ca\") pod \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") "
Oct 31 18:32:16.242358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.242297    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-openshift-ca\") pod \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\" (UID: \"105763cd-fdcc-434b-a946-dfcfbdcc5e2e\") "
Oct 31 18:32:16.242775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.242737    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-openshift-ca" pod "105763cd-fdcc-434b-a946-dfcfbdcc5e2e" (UID: "105763cd-fdcc-434b-a946-dfcfbdcc5e2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:16.244840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.244813    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-xks-ca" pod "105763cd-fdcc-434b-a946-dfcfbdcc5e2e" (UID: "105763cd-fdcc-434b-a946-dfcfbdcc5e2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:16.245009 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.244987    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-kube-api-access-s6mbx" pod "105763cd-fdcc-434b-a946-dfcfbdcc5e2e" (UID: "105763cd-fdcc-434b-a946-dfcfbdcc5e2e"). InnerVolumeSpecName "kube-api-access-s6mbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:16.343465 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.343437    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:16.343617 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.343496    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:16.343617 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.343508    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mbx\" (UniqueName: \"kubernetes.io/projected/105763cd-fdcc-434b-a946-dfcfbdcc5e2e-kube-api-access-s6mbx\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:16.604061 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:16.603974    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:32:16.994261 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.994229    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"
Oct 31 18:32:16.994438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.994259    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs" event={"ID":"105763cd-fdcc-434b-a946-dfcfbdcc5e2e","Type":"ContainerDied","Data":"f89738fc13f1d6174b6b3659c0bac10a9efe4786d30704f04bb1d6025422ffc7"}
Oct 31 18:32:16.994438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:16.994313    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89738fc13f1d6174b6b3659c0bac10a9efe4786d30704f04bb1d6025422ffc7"
Oct 31 18:32:22.969743 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:22.969709    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"]
Oct 31 18:32:23.020089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:23.020064    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4c133ca-m9n45"]
Oct 31 18:32:24.909427 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:24.909395    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e499f0-111d-443b-ac68-da3056572047" path="/var/lib/kubelet/pods/b8e499f0-111d-443b-ac68-da3056572047/volumes"
Oct 31 18:32:26.865679 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:26.865636    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"]
Oct 31 18:32:26.975548 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:26.975513    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"]
Oct 31 18:32:29.283372 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:29.283330    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"]
Oct 31 18:32:29.331522 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:29.331488    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bbf5caaf-vktcs"]
Oct 31 18:32:30.753196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:30.753167    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"]
Oct 31 18:32:30.753638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:30.753398    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy" containerID="cri-o://0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc" gracePeriod=30
Oct 31 18:32:30.909524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:30.909478    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105763cd-fdcc-434b-a946-dfcfbdcc5e2e" path="/var/lib/kubelet/pods/105763cd-fdcc-434b-a946-dfcfbdcc5e2e/volumes"
Oct 31 18:32:31.647375 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:31.647340    5551 patch_prober.go:29] interesting pod/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:32:31.647574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:31.647410    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:32:31.866042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:31.866003    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw" podUID="5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310" containerName="payload-processing" containerID="cri-o://050913234ea3bc41b4925f9593baa653b14fedb9ffa23667e5978e0f13a36447" gracePeriod=25
Oct 31 18:32:31.976523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:31.976430    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5" podUID="6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8" containerName="payload-pre-processing" containerID="cri-o://3679f7951f9361133d9ca9f926c05d29b075d265e66f95903dab4852088f77a5" gracePeriod=25
Oct 31 18:32:32.103030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.102751    5551 generic.go:356] "Generic (PLEG): container finished" podID="5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310" containerID="050913234ea3bc41b4925f9593baa653b14fedb9ffa23667e5978e0f13a36447" exitCode=0
Oct 31 18:32:32.103030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.102955    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw" event={"ID":"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310","Type":"ContainerDied","Data":"050913234ea3bc41b4925f9593baa653b14fedb9ffa23667e5978e0f13a36447"}
Oct 31 18:32:32.104880 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.104854    5551 generic.go:356] "Generic (PLEG): container finished" podID="6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8" containerID="3679f7951f9361133d9ca9f926c05d29b075d265e66f95903dab4852088f77a5" exitCode=0
Oct 31 18:32:32.104987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.104890    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5" event={"ID":"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8","Type":"ContainerDied","Data":"3679f7951f9361133d9ca9f926c05d29b075d265e66f95903dab4852088f77a5"}
Oct 31 18:32:32.129976 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.129946    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:32:32.249265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.247650    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-kube-api-access-fdfcq\" (UniqueName: \"kubernetes.io/projected/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-kube-api-access-fdfcq\") pod \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\" (UID: \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\") "
Oct 31 18:32:32.249265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.247829    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-plugins-config-volume\") pod \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\" (UID: \"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310\") "
Oct 31 18:32:32.249265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.248503    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-plugins-config-volume" pod "5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310" (UID: "5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:32.250885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.250856    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-kube-api-access-fdfcq" pod "5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310" (UID: "5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310"). InnerVolumeSpecName "kube-api-access-fdfcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:32.278851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.278823    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:32:32.349250 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.349216    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-kube-api-access-k26ts\" (UniqueName: \"kubernetes.io/projected/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-kube-api-access-k26ts\") pod \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\" (UID: \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\") "
Oct 31 18:32:32.349623 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.349484    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-plugins-config-volume\") pod \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\" (UID: \"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8\") "
Oct 31 18:32:32.352490 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.350142    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdfcq\" (UniqueName: \"kubernetes.io/projected/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-kube-api-access-fdfcq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:32.352490 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.350193    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:32.352490 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.350439    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-plugins-config-volume" pod "6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8" (UID: "6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:32.355392 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.355359    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-kube-api-access-k26ts" pod "6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8" (UID: "6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8"). InnerVolumeSpecName "kube-api-access-k26ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:32.451223 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.451195    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:32.451223 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:32.451220    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26ts\" (UniqueName: \"kubernetes.io/projected/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8-kube-api-access-k26ts\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:33.112826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.112797    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"
Oct 31 18:32:33.113310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.112838    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5" event={"ID":"6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8","Type":"ContainerDied","Data":"da3946a41a12b7c820b9070e93ec497384e2bf510bdeaaace6cbeddb2427aba4"}
Oct 31 18:32:33.113310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.112890    5551 scope.go:122] "RemoveContainer" containerID="3679f7951f9361133d9ca9f926c05d29b075d265e66f95903dab4852088f77a5"
Oct 31 18:32:33.114578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.114559    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"
Oct 31 18:32:33.114774 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.114581    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw" event={"ID":"5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310","Type":"ContainerDied","Data":"262e10303191df90c7f58a1e3dbec85f6644e8d8b8dc5706dc8b2af495ce125b"}
Oct 31 18:32:33.123766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.123749    5551 scope.go:122] "RemoveContainer" containerID="050913234ea3bc41b4925f9593baa653b14fedb9ffa23667e5978e0f13a36447"
Oct 31 18:32:33.277995 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.277957    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"]
Oct 31 18:32:33.331894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.331822    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1f4b473d-5844dcb995-bcntw"]
Oct 31 18:32:33.405153 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.405055    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"]
Oct 31 18:32:33.411243 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:33.411214    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1f4b473d-7bd8b97999-sxlw5"]
Oct 31 18:32:34.620396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.620367    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"]
Oct 31 18:32:34.620838 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.620689    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc" podUID="0f53dc43-7512-4520-ac0c-542ece3c6300" containerName="maas-api" containerID="cri-o://e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8" gracePeriod=30
Oct 31 18:32:34.896613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.896582    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:32:34.909204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.909178    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310" path="/var/lib/kubelet/pods/5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310/volumes"
Oct 31 18:32:34.909748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.909730    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8" path="/var/lib/kubelet/pods/6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8/volumes"
Oct 31 18:32:34.980684 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.980657    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-metrics-certs\") pod \"0f53dc43-7512-4520-ac0c-542ece3c6300\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") "
Oct 31 18:32:34.980835 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.980706    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f53dc43-7512-4520-ac0c-542ece3c6300-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53dc43-7512-4520-ac0c-542ece3c6300-openshift-service-ca\") pod \"0f53dc43-7512-4520-ac0c-542ece3c6300\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") "
Oct 31 18:32:34.980835 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.980779    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-maas-api-tls\") pod \"0f53dc43-7512-4520-ac0c-542ece3c6300\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") "
Oct 31 18:32:34.980976 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.980885    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f53dc43-7512-4520-ac0c-542ece3c6300-kube-api-access-v4btt\" (UniqueName: \"kubernetes.io/projected/0f53dc43-7512-4520-ac0c-542ece3c6300-kube-api-access-v4btt\") pod \"0f53dc43-7512-4520-ac0c-542ece3c6300\" (UID: \"0f53dc43-7512-4520-ac0c-542ece3c6300\") "
Oct 31 18:32:34.981146 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.981106    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f53dc43-7512-4520-ac0c-542ece3c6300-openshift-service-ca" pod "0f53dc43-7512-4520-ac0c-542ece3c6300" (UID: "0f53dc43-7512-4520-ac0c-542ece3c6300"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:34.981275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.981250    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53dc43-7512-4520-ac0c-542ece3c6300-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:34.983128 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.983102    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-metrics-certs" pod "0f53dc43-7512-4520-ac0c-542ece3c6300" (UID: "0f53dc43-7512-4520-ac0c-542ece3c6300"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:34.983227 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.983192    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f53dc43-7512-4520-ac0c-542ece3c6300-kube-api-access-v4btt" pod "0f53dc43-7512-4520-ac0c-542ece3c6300" (UID: "0f53dc43-7512-4520-ac0c-542ece3c6300"). InnerVolumeSpecName "kube-api-access-v4btt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:34.983267 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:34.983247    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-maas-api-tls" pod "0f53dc43-7512-4520-ac0c-542ece3c6300" (UID: "0f53dc43-7512-4520-ac0c-542ece3c6300"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:35.082211 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.082186    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:35.082211 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.082209    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53dc43-7512-4520-ac0c-542ece3c6300-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:35.082396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.082228    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4btt\" (UniqueName: \"kubernetes.io/projected/0f53dc43-7512-4520-ac0c-542ece3c6300-kube-api-access-v4btt\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:35.131433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.131403    5551 generic.go:356] "Generic (PLEG): container finished" podID="0f53dc43-7512-4520-ac0c-542ece3c6300" containerID="e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8" exitCode=0
Oct 31 18:32:35.131615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.131470    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"
Oct 31 18:32:35.131615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.131491    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc" event={"ID":"0f53dc43-7512-4520-ac0c-542ece3c6300","Type":"ContainerDied","Data":"e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8"}
Oct 31 18:32:35.131615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.131542    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc" event={"ID":"0f53dc43-7512-4520-ac0c-542ece3c6300","Type":"ContainerDied","Data":"9cec089898f23598f056161dd03e22a7f578df118c0ce65f4c793f43711a47b6"}
Oct 31 18:32:35.131615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.131571    5551 scope.go:122] "RemoveContainer" containerID="e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8"
Oct 31 18:32:35.143942 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.142573    5551 scope.go:122] "RemoveContainer" containerID="e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8"
Oct 31 18:32:35.144043 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:35.144016    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8\": container with ID starting with e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8 not found: ID does not exist" containerID="e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8"
Oct 31 18:32:35.144086 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.144056    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8"} err="failed to get container status \"e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8\": rpc error: code = NotFound desc = could not find container \"e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8\": container with ID starting with e17fd7648a613033d36a6b26edb8f5023087bab922b8655a765a2a17cd288ed8 not found: ID does not exist"
Oct 31 18:32:35.176178 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.176153    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"]
Oct 31 18:32:35.229330 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.229309    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"]
Oct 31 18:32:35.296623 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.296595    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"]
Oct 31 18:32:35.298633 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:35.298615    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c7a8371-7cd4b9798d-gwgwc"]
Oct 31 18:32:36.008195 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.008172    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:32:36.098817 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.098750    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-token\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-token\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.098817 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.098797    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c51df8e-18c3-4017-a30e-c4d895fec593-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c51df8e-18c3-4017-a30e-c4d895fec593-istiod-ca-cert\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.098974 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.098820    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-certs\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.098974 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.098848    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-podinfo\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.098974 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.098870    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-socket\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.098974 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.098892    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-credential-socket\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.098974 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.098961    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-data\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.099213 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.098997    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-kube-api-access-jmrnp\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-kube-api-access-jmrnp\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.099213 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099049    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-envoy\") pod \"4c51df8e-18c3-4017-a30e-c4d895fec593\" (UID: \"4c51df8e-18c3-4017-a30e-c4d895fec593\") "
Oct 31 18:32:36.099316 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099202    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-socket" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:36.099316 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099247    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-credential-socket" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:36.099316 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099263    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-data" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:36.099316 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099271    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c51df8e-18c3-4017-a30e-c4d895fec593-istiod-ca-cert" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:36.099316 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099278    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-certs" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:36.099626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099548    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.099626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099577    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c51df8e-18c3-4017-a30e-c4d895fec593-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.099626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099593    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.099626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099606    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.099626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.099621    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.101347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.101322    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-token" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:36.101347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.101333    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-kube-api-access-jmrnp" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "kube-api-access-jmrnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:36.101347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.101343    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-podinfo" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:32:36.101347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.101331    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-envoy" pod "4c51df8e-18c3-4017-a30e-c4d895fec593" (UID: "4c51df8e-18c3-4017-a30e-c4d895fec593"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:36.140965 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.140931    5551 generic.go:356] "Generic (PLEG): container finished" podID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerID="0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc" exitCode=0
Oct 31 18:32:36.141148 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.140994    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"
Oct 31 18:32:36.141148 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.140993    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" event={"ID":"4c51df8e-18c3-4017-a30e-c4d895fec593","Type":"ContainerDied","Data":"0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc"}
Oct 31 18:32:36.141148 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.141055    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c" event={"ID":"4c51df8e-18c3-4017-a30e-c4d895fec593","Type":"ContainerDied","Data":"dbc80c335227746391d5f4c70736a64e5640d196b281a84b6d66e78d0d7a7c7c"}
Oct 31 18:32:36.141148 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.141088    5551 scope.go:122] "RemoveContainer" containerID="0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc"
Oct 31 18:32:36.152205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.152182    5551 scope.go:122] "RemoveContainer" containerID="3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126"
Oct 31 18:32:36.161864 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.161829    5551 scope.go:122] "RemoveContainer" containerID="0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc"
Oct 31 18:32:36.162222 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:36.162174    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc\": container with ID starting with 0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc not found: ID does not exist" containerID="0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc"
Oct 31 18:32:36.162329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.162212    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc"} err="failed to get container status \"0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc\": rpc error: code = NotFound desc = could not find container \"0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc\": container with ID starting with 0164a917544033267e3b89ff26e09c7e7186011acdd883cd6028eca9bdd1becc not found: ID does not exist"
Oct 31 18:32:36.162329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.162238    5551 scope.go:122] "RemoveContainer" containerID="3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126"
Oct 31 18:32:36.162493 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:36.162448    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126\": container with ID starting with 3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126 not found: ID does not exist" containerID="3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126"
Oct 31 18:32:36.162543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.162506    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126"} err="failed to get container status \"3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126\": rpc error: code = NotFound desc = could not find container \"3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126\": container with ID starting with 3f7774cba22c9815026e7616cf2969c438a04a2491379f8939b7363453224126 not found: ID does not exist"
Oct 31 18:32:36.200731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.200695    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.200731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.200725    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.200731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.200735    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrnp\" (UniqueName: \"kubernetes.io/projected/4c51df8e-18c3-4017-a30e-c4d895fec593-kube-api-access-jmrnp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.200972 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.200748    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c51df8e-18c3-4017-a30e-c4d895fec593-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:36.305593 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.305562    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"]
Oct 31 18:32:36.308903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.308878    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bbf5caaf-openshift-default-7476c4748f-9nn5c"]
Oct 31 18:32:36.909116 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.909087    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f53dc43-7512-4520-ac0c-542ece3c6300" path="/var/lib/kubelet/pods/0f53dc43-7512-4520-ac0c-542ece3c6300/volumes"
Oct 31 18:32:36.909521 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:36.909507    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" path="/var/lib/kubelet/pods/4c51df8e-18c3-4017-a30e-c4d895fec593/volumes"
Oct 31 18:32:40.176412 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.176360    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl" podUID="b28190c0-6d04-4edd-95a1-a6646791ce7d" containerName="payload-processing" containerID="cri-o://74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1" gracePeriod=25
Oct 31 18:32:40.229955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.229886    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc" podUID="2399694e-1948-40ed-bb1c-5220115f72d1" containerName="payload-pre-processing" containerID="cri-o://d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264" gracePeriod=25
Oct 31 18:32:40.478608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.478582    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:32:40.507101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.507080    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:32:40.541965 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.541937    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b28190c0-6d04-4edd-95a1-a6646791ce7d-kube-api-access-rzv2t\" (UniqueName: \"kubernetes.io/projected/b28190c0-6d04-4edd-95a1-a6646791ce7d-kube-api-access-rzv2t\") pod \"b28190c0-6d04-4edd-95a1-a6646791ce7d\" (UID: \"b28190c0-6d04-4edd-95a1-a6646791ce7d\") "
Oct 31 18:32:40.542101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.542059    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b28190c0-6d04-4edd-95a1-a6646791ce7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28190c0-6d04-4edd-95a1-a6646791ce7d-plugins-config-volume\") pod \"b28190c0-6d04-4edd-95a1-a6646791ce7d\" (UID: \"b28190c0-6d04-4edd-95a1-a6646791ce7d\") "
Oct 31 18:32:40.542394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.542368    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28190c0-6d04-4edd-95a1-a6646791ce7d-plugins-config-volume" pod "b28190c0-6d04-4edd-95a1-a6646791ce7d" (UID: "b28190c0-6d04-4edd-95a1-a6646791ce7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:40.544106 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.544085    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28190c0-6d04-4edd-95a1-a6646791ce7d-kube-api-access-rzv2t" pod "b28190c0-6d04-4edd-95a1-a6646791ce7d" (UID: "b28190c0-6d04-4edd-95a1-a6646791ce7d"). InnerVolumeSpecName "kube-api-access-rzv2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:40.643608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.643580    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2399694e-1948-40ed-bb1c-5220115f72d1-kube-api-access-b2vh4\" (UniqueName: \"kubernetes.io/projected/2399694e-1948-40ed-bb1c-5220115f72d1-kube-api-access-b2vh4\") pod \"2399694e-1948-40ed-bb1c-5220115f72d1\" (UID: \"2399694e-1948-40ed-bb1c-5220115f72d1\") "
Oct 31 18:32:40.643771 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.643712    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2399694e-1948-40ed-bb1c-5220115f72d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399694e-1948-40ed-bb1c-5220115f72d1-plugins-config-volume\") pod \"2399694e-1948-40ed-bb1c-5220115f72d1\" (UID: \"2399694e-1948-40ed-bb1c-5220115f72d1\") "
Oct 31 18:32:40.644090 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.644072    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28190c0-6d04-4edd-95a1-a6646791ce7d-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:40.644138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.644099    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzv2t\" (UniqueName: \"kubernetes.io/projected/b28190c0-6d04-4edd-95a1-a6646791ce7d-kube-api-access-rzv2t\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:40.644138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.644106    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2399694e-1948-40ed-bb1c-5220115f72d1-plugins-config-volume" pod "2399694e-1948-40ed-bb1c-5220115f72d1" (UID: "2399694e-1948-40ed-bb1c-5220115f72d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:40.645807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.645787    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2399694e-1948-40ed-bb1c-5220115f72d1-kube-api-access-b2vh4" pod "2399694e-1948-40ed-bb1c-5220115f72d1" (UID: "2399694e-1948-40ed-bb1c-5220115f72d1"). InnerVolumeSpecName "kube-api-access-b2vh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:40.745434 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.745362    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2vh4\" (UniqueName: \"kubernetes.io/projected/2399694e-1948-40ed-bb1c-5220115f72d1-kube-api-access-b2vh4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:40.745434 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:40.745388    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2399694e-1948-40ed-bb1c-5220115f72d1-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:41.189842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.189804    5551 generic.go:356] "Generic (PLEG): container finished" podID="2399694e-1948-40ed-bb1c-5220115f72d1" containerID="d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264" exitCode=0
Oct 31 18:32:41.190335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.189875    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"
Oct 31 18:32:41.190335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.189891    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc" event={"ID":"2399694e-1948-40ed-bb1c-5220115f72d1","Type":"ContainerDied","Data":"d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264"}
Oct 31 18:32:41.190335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.189980    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc" event={"ID":"2399694e-1948-40ed-bb1c-5220115f72d1","Type":"ContainerDied","Data":"887dbc39b4e14f0d50eaa288ed58131ab18f5fd17ece2abe8c07b05258abc43d"}
Oct 31 18:32:41.190335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.190009    5551 scope.go:122] "RemoveContainer" containerID="d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264"
Oct 31 18:32:41.191541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.191516    5551 generic.go:356] "Generic (PLEG): container finished" podID="b28190c0-6d04-4edd-95a1-a6646791ce7d" containerID="74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1" exitCode=0
Oct 31 18:32:41.191630 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.191572    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl" event={"ID":"b28190c0-6d04-4edd-95a1-a6646791ce7d","Type":"ContainerDied","Data":"74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1"}
Oct 31 18:32:41.191630 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.191598    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"
Oct 31 18:32:41.191726 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.191635    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl" event={"ID":"b28190c0-6d04-4edd-95a1-a6646791ce7d","Type":"ContainerDied","Data":"b332c7df7fc55ec8555676819c3c99e5be13e82ef3b460e5f067b93c90b916df"}
Oct 31 18:32:41.201856 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.201838    5551 scope.go:122] "RemoveContainer" containerID="d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264"
Oct 31 18:32:41.202221 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:41.202200    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264\": container with ID starting with d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264 not found: ID does not exist" containerID="d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264"
Oct 31 18:32:41.202269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.202233    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264"} err="failed to get container status \"d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264\": rpc error: code = NotFound desc = could not find container \"d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264\": container with ID starting with d01f28827c001dc3f14ea86d2bc2e260ef262975b96dfa0eb5ad235ddea43264 not found: ID does not exist"
Oct 31 18:32:41.202269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.202252    5551 scope.go:122] "RemoveContainer" containerID="74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1"
Oct 31 18:32:41.211547 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.211527    5551 scope.go:122] "RemoveContainer" containerID="74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1"
Oct 31 18:32:41.211844 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:41.211825    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1\": container with ID starting with 74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1 not found: ID does not exist" containerID="74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1"
Oct 31 18:32:41.211915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.211854    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1"} err="failed to get container status \"74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1\": rpc error: code = NotFound desc = could not find container \"74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1\": container with ID starting with 74009fb1f17fd1fae85e08d640b391875b00caadd8b3f2eeb37cfef9db5716a1 not found: ID does not exist"
Oct 31 18:32:41.354477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.354425    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"]
Oct 31 18:32:41.403970 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.403935    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c7a8371-559f667689-rxtbl"]
Oct 31 18:32:41.464862 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.464775    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"]
Oct 31 18:32:41.466851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.466830    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c7a8371-54896cbf75-fplhc"]
Oct 31 18:32:41.693687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.693658    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"]
Oct 31 18:32:41.694129 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694115    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy"
Oct 31 18:32:41.694173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694133    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy"
Oct 31 18:32:41.694173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694155    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b28190c0-6d04-4edd-95a1-a6646791ce7d" containerName="payload-processing"
Oct 31 18:32:41.694173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694161    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b28190c0-6d04-4edd-95a1-a6646791ce7d" containerName="payload-processing"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694177    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8" containerName="payload-pre-processing"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694182    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8" containerName="payload-pre-processing"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694191    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310" containerName="payload-processing"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694195    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310" containerName="payload-processing"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694202    5551 cpu_manager.go:414] "containerMap: removing container" podUID="105763cd-fdcc-434b-a946-dfcfbdcc5e2e" containerName="revoke-keys"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694206    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105763cd-fdcc-434b-a946-dfcfbdcc5e2e" containerName="revoke-keys"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694217    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0f53dc43-7512-4520-ac0c-542ece3c6300" containerName="maas-api"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694224    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f53dc43-7512-4520-ac0c-542ece3c6300" containerName="maas-api"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694237    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2399694e-1948-40ed-bb1c-5220115f72d1" containerName="payload-pre-processing"
Oct 31 18:32:41.694265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694241    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2399694e-1948-40ed-bb1c-5220115f72d1" containerName="payload-pre-processing"
Oct 31 18:32:41.694561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694313    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2399694e-1948-40ed-bb1c-5220115f72d1" containerName="payload-pre-processing"
Oct 31 18:32:41.694561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694323    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy"
Oct 31 18:32:41.694561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694329    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b28190c0-6d04-4edd-95a1-a6646791ce7d" containerName="payload-processing"
Oct 31 18:32:41.694561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694337    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4edc17-6e21-40fe-bbf2-a2bd5a6e18f8" containerName="payload-pre-processing"
Oct 31 18:32:41.694561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694345    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f13d7b4-5f22-4a9d-bf6e-ae97fc40e310" containerName="payload-processing"
Oct 31 18:32:41.694561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694349    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="105763cd-fdcc-434b-a946-dfcfbdcc5e2e" containerName="revoke-keys"
Oct 31 18:32:41.694561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.694355    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f53dc43-7512-4520-ac0c-542ece3c6300" containerName="maas-api"
Oct 31 18:32:41.698688 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.698668    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.755177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.754613    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-88085e-serving-cert\""
Oct 31 18:32:41.755177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.754707    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 18:32:41.755177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.754879    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 18:32:41.755177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.754955    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 18:32:41.755786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.755740    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 18:32:41.758375 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.758344    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"]
Oct 31 18:32:41.856012 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.855974    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zl94\" (UniqueName: \"kubernetes.io/projected/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-kube-api-access-8zl94\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.856175 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.856046    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.856175 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.856126    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.856292 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.856246    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-metrics-certs\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.957854 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.957816    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zl94\" (UniqueName: \"kubernetes.io/projected/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-kube-api-access-8zl94\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.958150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.958119    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.958290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.958219    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.958358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.958327    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-metrics-certs\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.960404 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:41.959101    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-serving-cert: secret "maas-api-e2e-worker-w2-88085e-serving-cert" not found
Oct 31 18:32:41.960404 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:41.959199    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls podName:4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d nodeName:}" failed. No retries permitted until 2026-10-31 18:32:42.459159827 +0000 UTC m=+665.690398100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls") pod "maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n" (UID: "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d") : secret "maas-api-e2e-worker-w2-88085e-serving-cert" not found
Oct 31 18:32:41.960404 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.960314    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:41.962311 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:41.962287    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-metrics-certs\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:42.018278 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:42.018207    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zl94\" (UniqueName: \"kubernetes.io/projected/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-kube-api-access-8zl94\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:42.463860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:42.463828    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:42.466302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:42.466285    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:42.611267 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:42.611232    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:42.908006 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:42.907975    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2399694e-1948-40ed-bb1c-5220115f72d1" path="/var/lib/kubelet/pods/2399694e-1948-40ed-bb1c-5220115f72d1/volumes"
Oct 31 18:32:42.908501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:42.908480    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b28190c0-6d04-4edd-95a1-a6646791ce7d" path="/var/lib/kubelet/pods/b28190c0-6d04-4edd-95a1-a6646791ce7d/volumes"
Oct 31 18:32:42.991484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:42.991438    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"]
Oct 31 18:32:42.993424 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:32:42.993395    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bee3783_8dcd_42c6_86aa_abdd7ff0ae8d.slice/crio-6f0fd24b296e858cce2dfd1cded27997b282d25cbfe15e495a7a3dd3de2564d1 WatchSource:0}: Error finding container 6f0fd24b296e858cce2dfd1cded27997b282d25cbfe15e495a7a3dd3de2564d1: Status 404 returned error can't find the container with id 6f0fd24b296e858cce2dfd1cded27997b282d25cbfe15e495a7a3dd3de2564d1
Oct 31 18:32:43.052144 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:43.052107    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"]
Oct 31 18:32:43.052397 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:43.052367    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy" containerID="cri-o://108ddb898917e4254d5cf2ef40fc7a14f9db8b0ce6911d85fb0d23acc6f7beab" gracePeriod=30
Oct 31 18:32:43.206839 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:43.206756    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n" event={"ID":"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d","Type":"ContainerStarted","Data":"6f0fd24b296e858cce2dfd1cded27997b282d25cbfe15e495a7a3dd3de2564d1"}
Oct 31 18:32:44.219749 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.219705    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n" event={"ID":"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d","Type":"ContainerStarted","Data":"bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb"}
Oct 31 18:32:44.220144 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.219815    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:44.334921 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.334865    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n" podStartSLOduration=2.4748827110000002 podStartE2EDuration="3.334849089s" totalImagesPullingTime="859.966378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:44.333591456 +0000 UTC m=+667.564829747" watchObservedRunningTime="2026-10-31 18:32:44.334849089 +0000 UTC m=+667.566087378"
Oct 31 18:32:44.419069 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.419034    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"]
Oct 31 18:32:44.419779 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.419757    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy"
Oct 31 18:32:44.419779 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.419780    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy"
Oct 31 18:32:44.419966 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.419890    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c51df8e-18c3-4017-a30e-c4d895fec593" containerName="istio-proxy"
Oct 31 18:32:44.423188 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.423172    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.485573 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.481392    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-da6ea7-openshift-default-dockercfg-z7k66\""
Oct 31 18:32:44.485740 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.485710    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"]
Oct 31 18:32:44.603064 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603029    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.603216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603070    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.603216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603113    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.603309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603224    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcwsb\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-kube-api-access-rcwsb\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.603309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603260    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.603393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603363    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.603439 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603396    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.603522 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603508    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.603556 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.603533    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.704723 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.704680    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.704940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.704733    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.704940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.704784    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.704940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.704853    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcwsb\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-kube-api-access-rcwsb\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.704940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.704890    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.705193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.704969    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.705193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.705009    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.705193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.705094    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.705193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.705131    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.705505 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.705444    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.705604 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.705551    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.705768 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.705741    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.706041 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.705987    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.706041 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.706023    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.708622 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.708597    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.708903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.708880    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.761298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.761239    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:44.761592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:44.761568    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcwsb\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-kube-api-access-rcwsb\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:45.037629 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.037548    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:45.463522 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.463483    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"]
Oct 31 18:32:45.516649 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:32:45.515626    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0e9132_e0de_4ad8_bc9a_0e621973e643.slice/crio-e9cccab912d1b75149120a2f3bd9ace24b7138d610d06fb34ff6b7a865cf96a7 WatchSource:0}: Error finding container e9cccab912d1b75149120a2f3bd9ace24b7138d610d06fb34ff6b7a865cf96a7: Status 404 returned error can't find the container with id e9cccab912d1b75149120a2f3bd9ace24b7138d610d06fb34ff6b7a865cf96a7
Oct 31 18:32:45.519199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.519162    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:45.519328 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.519253    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:45.519328 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.519302    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:45.642841 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.642806    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"]
Oct 31 18:32:45.646478 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.646438    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:45.700300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.700262    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-6279f5\""
Oct 31 18:32:45.700871 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.700628    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-6279f5-dockercfg-x5vh6\""
Oct 31 18:32:45.703030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.703003    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"]
Oct 31 18:32:45.734565 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.733853    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46532445-f525-4dfd-ac76-c0d2e7c26f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr\" (UID: \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:45.734565 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.734021    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqbn\" (UniqueName: \"kubernetes.io/projected/46532445-f525-4dfd-ac76-c0d2e7c26f2d-kube-api-access-jwqbn\") pod \"payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr\" (UID: \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:45.834737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.834669    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46532445-f525-4dfd-ac76-c0d2e7c26f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr\" (UID: \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:45.834946 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.834811    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqbn\" (UniqueName: \"kubernetes.io/projected/46532445-f525-4dfd-ac76-c0d2e7c26f2d-kube-api-access-jwqbn\") pod \"payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr\" (UID: \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:45.835443 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.835417    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46532445-f525-4dfd-ac76-c0d2e7c26f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr\" (UID: \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:45.891128 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.891102    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqbn\" (UniqueName: \"kubernetes.io/projected/46532445-f525-4dfd-ac76-c0d2e7c26f2d-kube-api-access-jwqbn\") pod \"payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr\" (UID: \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:45.961557 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:45.961528    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:46.232735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:46.232701    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" event={"ID":"9a0e9132-e0de-4ad8-bc9a-0e621973e643","Type":"ContainerStarted","Data":"ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b"}
Oct 31 18:32:46.232903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:46.232753    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" event={"ID":"9a0e9132-e0de-4ad8-bc9a-0e621973e643","Type":"ContainerStarted","Data":"e9cccab912d1b75149120a2f3bd9ace24b7138d610d06fb34ff6b7a865cf96a7"}
Oct 31 18:32:46.331763 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:46.331649    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"]
Oct 31 18:32:46.334433 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:32:46.334399    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46532445_f525_4dfd_ac76_c0d2e7c26f2d.slice/crio-845738a272e14ee700cb6ffc8e0cc47b45f0402638ec171056f5bfdcddc16fe3 WatchSource:0}: Error finding container 845738a272e14ee700cb6ffc8e0cc47b45f0402638ec171056f5bfdcddc16fe3: Status 404 returned error can't find the container with id 845738a272e14ee700cb6ffc8e0cc47b45f0402638ec171056f5bfdcddc16fe3
Oct 31 18:32:46.349221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:46.349168    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" podStartSLOduration=2.349151302 podStartE2EDuration="2.349151302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:46.346506228 +0000 UTC m=+669.577744518" watchObservedRunningTime="2026-10-31 18:32:46.349151302 +0000 UTC m=+669.580389592"
Oct 31 18:32:46.604473 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:46.604373    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:32:47.038544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:47.038510    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:47.044918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:47.044895    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:47.240433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:47.240398    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr" event={"ID":"46532445-f525-4dfd-ac76-c0d2e7c26f2d","Type":"ContainerStarted","Data":"51700b711015c9d9e83c5b2fb3fcb0413538996367e448df69723ec745e26cbb"}
Oct 31 18:32:47.240626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:47.240450    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr" event={"ID":"46532445-f525-4dfd-ac76-c0d2e7c26f2d","Type":"ContainerStarted","Data":"845738a272e14ee700cb6ffc8e0cc47b45f0402638ec171056f5bfdcddc16fe3"}
Oct 31 18:32:47.240771 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:47.240740    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:47.240852 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:47.240785    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:32:47.241745 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:47.241727    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:32:47.378597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:47.378542    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr" podStartSLOduration=2.37852656 podStartE2EDuration="2.37852656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:47.374805438 +0000 UTC m=+670.606043727" watchObservedRunningTime="2026-10-31 18:32:47.37852656 +0000 UTC m=+670.609764849"
Oct 31 18:32:48.250827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.250800    5551 generic.go:356] "Generic (PLEG): container finished" podID="73976601-d3aa-41b4-a77e-eea3774707ac" containerID="108ddb898917e4254d5cf2ef40fc7a14f9db8b0ce6911d85fb0d23acc6f7beab" exitCode=0
Oct 31 18:32:48.251241 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.250869    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" event={"ID":"73976601-d3aa-41b4-a77e-eea3774707ac","Type":"ContainerDied","Data":"108ddb898917e4254d5cf2ef40fc7a14f9db8b0ce6911d85fb0d23acc6f7beab"}
Oct 31 18:32:48.251241 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.250930    5551 scope.go:122] "RemoveContainer" containerID="0b0f19814d873e1199f3a488fffeaf60480303de40069864696694fd8fd2cea3"
Oct 31 18:32:48.316160 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.316141    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:32:48.373298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373124    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-envoy\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373170    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-credential-socket\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373261    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-data\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373294    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-certs\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373342    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-socket\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373375    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73976601-d3aa-41b4-a77e-eea3774707ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73976601-d3aa-41b4-a77e-eea3774707ac-istio-podinfo\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373404    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-kube-api-access-z8xxj\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-kube-api-access-z8xxj\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373425    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73976601-d3aa-41b4-a77e-eea3774707ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73976601-d3aa-41b4-a77e-eea3774707ac-istiod-ca-cert\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373474    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-istio-token\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-istio-token\") pod \"73976601-d3aa-41b4-a77e-eea3774707ac\" (UID: \"73976601-d3aa-41b4-a77e-eea3774707ac\") "
Oct 31 18:32:48.373887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373748    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-certs" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:48.373887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.373811    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-credential-socket" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:48.374066 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.374032    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-socket" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:48.374182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.374092    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.374182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.374113    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.374295 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.374175    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73976601-d3aa-41b4-a77e-eea3774707ac-istiod-ca-cert" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:48.374649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.374629    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-data" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:48.376943 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.376913    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-kube-api-access-z8xxj" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "kube-api-access-z8xxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:48.377343 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.377322    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-istio-token" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:48.377664 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.377628    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73976601-d3aa-41b4-a77e-eea3774707ac-istio-podinfo" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:32:48.377904 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.377882    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-envoy" pod "73976601-d3aa-41b4-a77e-eea3774707ac" (UID: "73976601-d3aa-41b4-a77e-eea3774707ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:48.475112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.475073    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.475112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.475112    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73976601-d3aa-41b4-a77e-eea3774707ac-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.475333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.475127    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xxj\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-kube-api-access-z8xxj\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.475333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.475142    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73976601-d3aa-41b4-a77e-eea3774707ac-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.475333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.475156    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73976601-d3aa-41b4-a77e-eea3774707ac-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.475333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.475169    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.475333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.475182    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73976601-d3aa-41b4-a77e-eea3774707ac-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:48.737935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.737900    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"]
Oct 31 18:32:48.738228 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:48.738152    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy" containerID="cri-o://d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d" gracePeriod=30
Oct 31 18:32:49.259028 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:49.258997    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"
Oct 31 18:32:49.259442 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:49.259039    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9" event={"ID":"73976601-d3aa-41b4-a77e-eea3774707ac","Type":"ContainerDied","Data":"1ebddf258d1237e21cd4e2682589a0d8e4112b0808e01c29cb617811f97bfd0c"}
Oct 31 18:32:49.259442 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:49.259094    5551 scope.go:122] "RemoveContainer" containerID="108ddb898917e4254d5cf2ef40fc7a14f9db8b0ce6911d85fb0d23acc6f7beab"
Oct 31 18:32:49.424625 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:49.424588    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"]
Oct 31 18:32:49.427009 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:49.426983    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3c7a8371-openshift-default-c756669c-hl5q9"]
Oct 31 18:32:50.069382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.069341    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"]
Oct 31 18:32:50.069824 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.069812    5551 cpu_manager.go:414] "containerMap: removing container" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy"
Oct 31 18:32:50.069876 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.069826    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy"
Oct 31 18:32:50.069876 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.069850    5551 cpu_manager.go:414] "containerMap: removing container" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy"
Oct 31 18:32:50.069876 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.069855    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy"
Oct 31 18:32:50.069962 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.069916    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy"
Oct 31 18:32:50.069962 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.069927    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" containerName="istio-proxy"
Oct 31 18:32:50.074319 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.074296    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.126576 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.126545    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e4a2c-openshift-default-dockercfg-7jr6n\""
Oct 31 18:32:50.180343 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.180310    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"]
Oct 31 18:32:50.193815 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.193778    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.193996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.193823    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.193996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.193847    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972gs\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-kube-api-access-972gs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.193996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.193925    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.193996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.193976    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.194133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.194053    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.194133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.194115    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.194242 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.194136    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.194242 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.194181    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.231774 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.231747    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 18:32:50.294912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.294882    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.294922    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.294969    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295026    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295216    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295263    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295387    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295390    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295431    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295431    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295452    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972gs\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-kube-api-access-972gs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295700    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295817    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.295986 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.295944    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.297765 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.297741    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.297895 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.297876    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.361927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.361840    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.362084 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.361998    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972gs\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-kube-api-access-972gs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.387680 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.387649    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:50.817165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.817137    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"]
Oct 31 18:32:50.818741 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:32:50.818716    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ca5d99_8de5_497d_aae0_6e5a6536b739.slice/crio-ee9026668fd877b81e95937a7080e9f2a98d599c5208accd5284b52d70081fa6 WatchSource:0}: Error finding container ee9026668fd877b81e95937a7080e9f2a98d599c5208accd5284b52d70081fa6: Status 404 returned error can't find the container with id ee9026668fd877b81e95937a7080e9f2a98d599c5208accd5284b52d70081fa6
Oct 31 18:32:50.820923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.820892    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:50.821007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.820955    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:50.821007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.820982    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:32:50.915881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:50.915842    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73976601-d3aa-41b4-a77e-eea3774707ac" path="/var/lib/kubelet/pods/73976601-d3aa-41b4-a77e-eea3774707ac/volumes"
Oct 31 18:32:51.276634 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:51.276588    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" event={"ID":"c5ca5d99-8de5-497d-aae0-6e5a6536b739","Type":"ContainerStarted","Data":"c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d"}
Oct 31 18:32:51.276820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:51.276652    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" event={"ID":"c5ca5d99-8de5-497d-aae0-6e5a6536b739","Type":"ContainerStarted","Data":"ee9026668fd877b81e95937a7080e9f2a98d599c5208accd5284b52d70081fa6"}
Oct 31 18:32:51.388832 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:51.388779    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:51.394898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:51.394836    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" podStartSLOduration=1.394815657 podStartE2EDuration="1.394815657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:51.392101186 +0000 UTC m=+674.623339506" watchObservedRunningTime="2026-10-31 18:32:51.394815657 +0000 UTC m=+674.626053948"
Oct 31 18:32:51.395410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:51.395388    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:52.283101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:52.283071    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:52.284293 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:52.284270    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:32:53.203738 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.203701    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"]
Oct 31 18:32:53.207936 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.207918    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.265079 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.265046    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"]
Oct 31 18:32:53.311187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.311163    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-6279f5-serving-cert\""
Oct 31 18:32:53.340562 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.340532    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvlt\" (UniqueName: \"kubernetes.io/projected/b1993cb5-c116-4c47-be36-3893da5088c9-kube-api-access-dpvlt\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.340709 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.340601    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-metrics-certs\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.340785 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.340740    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.340830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.340788    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1993cb5-c116-4c47-be36-3893da5088c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.441559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.441531    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvlt\" (UniqueName: \"kubernetes.io/projected/b1993cb5-c116-4c47-be36-3893da5088c9-kube-api-access-dpvlt\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.441692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.441577    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-metrics-certs\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.441692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.441626    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.441692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.441647    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1993cb5-c116-4c47-be36-3893da5088c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.441815 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:53.441767    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-serving-cert: secret "maas-api-e2e-worker-w4-6279f5-serving-cert" not found
Oct 31 18:32:53.441858 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:53.441823    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls podName:b1993cb5-c116-4c47-be36-3893da5088c9 nodeName:}" failed. No retries permitted until 2026-10-31 18:32:53.9418088 +0000 UTC m=+677.173047069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls") pod "maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w" (UID: "b1993cb5-c116-4c47-be36-3893da5088c9") : secret "maas-api-e2e-worker-w4-6279f5-serving-cert" not found
Oct 31 18:32:53.442348 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.442330    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1993cb5-c116-4c47-be36-3893da5088c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.444172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.444147    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-metrics-certs\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.501241 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.501170    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvlt\" (UniqueName: \"kubernetes.io/projected/b1993cb5-c116-4c47-be36-3893da5088c9-kube-api-access-dpvlt\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.947272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.947229    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.949802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.949775    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:53.997786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:53.997761    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:32:54.048044 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.047985    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-socket\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048044 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048017    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-istio-token\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-istio-token\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048044 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048043    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-envoy\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048066    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-credential-socket\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048119    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-data\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048151    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-certs\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048222    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-kube-api-access-lm9fl\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-kube-api-access-lm9fl\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048251    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-socket" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:54.048585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048322    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d073d45-50fa-48a2-b95d-09d666a92991-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d073d45-50fa-48a2-b95d-09d666a92991-istio-podinfo\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048376    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d073d45-50fa-48a2-b95d-09d666a92991-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d073d45-50fa-48a2-b95d-09d666a92991-istiod-ca-cert\") pod \"8d073d45-50fa-48a2-b95d-09d666a92991\" (UID: \"8d073d45-50fa-48a2-b95d-09d666a92991\") "
Oct 31 18:32:54.048585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048433    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-credential-socket" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:54.048585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048447    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-certs" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:54.048585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048484    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-data" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:54.049000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048778    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d073d45-50fa-48a2-b95d-09d666a92991-istiod-ca-cert" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:54.049000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048806    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.049000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048826    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.049000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048840    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.049000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.048850    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.050439 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.050420    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-istio-token" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:54.050571 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.050556    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-envoy" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:54.050989 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.050964    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d073d45-50fa-48a2-b95d-09d666a92991-istio-podinfo" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:32:54.050989 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.050966    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-kube-api-access-lm9fl" pod "8d073d45-50fa-48a2-b95d-09d666a92991" (UID: "8d073d45-50fa-48a2-b95d-09d666a92991"). InnerVolumeSpecName "kube-api-access-lm9fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:54.121927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.121902    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:54.149437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.149414    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9fl\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-kube-api-access-lm9fl\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.149552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.149441    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d073d45-50fa-48a2-b95d-09d666a92991-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.149552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.149475    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d073d45-50fa-48a2-b95d-09d666a92991-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.149552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.149489    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d073d45-50fa-48a2-b95d-09d666a92991-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.149552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.149502    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d073d45-50fa-48a2-b95d-09d666a92991-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:54.298481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.298377    5551 generic.go:356] "Generic (PLEG): container finished" podID="8d073d45-50fa-48a2-b95d-09d666a92991" containerID="d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d" exitCode=0
Oct 31 18:32:54.298481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.298418    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"
Oct 31 18:32:54.298950 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.298483    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" event={"ID":"8d073d45-50fa-48a2-b95d-09d666a92991","Type":"ContainerDied","Data":"d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d"}
Oct 31 18:32:54.298950 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.298551    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf" event={"ID":"8d073d45-50fa-48a2-b95d-09d666a92991","Type":"ContainerDied","Data":"1e1b01b10f255a547ae029d7290d802f805736f57896804fb4b8d169f244c6d8"}
Oct 31 18:32:54.298950 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.298587    5551 scope.go:122] "RemoveContainer" containerID="d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d"
Oct 31 18:32:54.308773 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.308750    5551 scope.go:122] "RemoveContainer" containerID="b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e"
Oct 31 18:32:54.318624 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.318604    5551 scope.go:122] "RemoveContainer" containerID="d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d"
Oct 31 18:32:54.318870 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:54.318853    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d\": container with ID starting with d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d not found: ID does not exist" containerID="d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d"
Oct 31 18:32:54.318911 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.318879    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d"} err="failed to get container status \"d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d\": rpc error: code = NotFound desc = could not find container \"d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d\": container with ID starting with d25983c18067bec3ead9e1d9eda970cbad186de29ce55bd9343f1de72898ab4d not found: ID does not exist"
Oct 31 18:32:54.318911 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.318896    5551 scope.go:122] "RemoveContainer" containerID="b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e"
Oct 31 18:32:54.319135 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:32:54.319121    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e\": container with ID starting with b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e not found: ID does not exist" containerID="b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e"
Oct 31 18:32:54.319186 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.319138    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e"} err="failed to get container status \"b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e\": rpc error: code = NotFound desc = could not find container \"b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e\": container with ID starting with b7a1b3e4f855dc949755c44f7e457c9e656428d0ef738c7e3e11841185e1353e not found: ID does not exist"
Oct 31 18:32:54.462364 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.462340    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"]
Oct 31 18:32:54.465875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.465852    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1f4b473d-gw-openshift-default-6bb769ddb9-tbgcf"]
Oct 31 18:32:54.498902 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.498873    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"]
Oct 31 18:32:54.539748 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:32:54.539713    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1993cb5_c116_4c47_be36_3893da5088c9.slice/crio-b4570defbfbaf2df8470b0ad674f0b530ff6fa05125446aa2af6bc94043cba83 WatchSource:0}: Error finding container b4570defbfbaf2df8470b0ad674f0b530ff6fa05125446aa2af6bc94043cba83: Status 404 returned error can't find the container with id b4570defbfbaf2df8470b0ad674f0b530ff6fa05125446aa2af6bc94043cba83
Oct 31 18:32:54.541131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.541113    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:32:54.910206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:54.910170    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" path="/var/lib/kubelet/pods/8d073d45-50fa-48a2-b95d-09d666a92991/volumes"
Oct 31 18:32:55.306050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:55.305977    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w" event={"ID":"b1993cb5-c116-4c47-be36-3893da5088c9","Type":"ContainerStarted","Data":"b4570defbfbaf2df8470b0ad674f0b530ff6fa05125446aa2af6bc94043cba83"}
Oct 31 18:32:56.314873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:56.314833    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w" event={"ID":"b1993cb5-c116-4c47-be36-3893da5088c9","Type":"ContainerStarted","Data":"8c6647a3d5707656c44d60af8621520b833ec88d533324b696522cb44a9b402e"}
Oct 31 18:32:56.315243 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:56.314924    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:32:56.432865 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:56.432816    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w" podStartSLOduration=2.642211906 podStartE2EDuration="3.432801185s" totalImagesPullingTime="790.589279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:56.431124408 +0000 UTC m=+679.662362724" watchObservedRunningTime="2026-10-31 18:32:56.432801185 +0000 UTC m=+679.664039475"
Oct 31 18:32:58.254548 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:32:58.254516    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 18:33:02.235547 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.235516    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"]
Oct 31 18:33:02.236225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.236204    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy"
Oct 31 18:33:02.236295 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.236230    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy"
Oct 31 18:33:02.236295 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.236259    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy"
Oct 31 18:33:02.236295 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.236265    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy"
Oct 31 18:33:02.236419 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.236351    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy"
Oct 31 18:33:02.239725 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.239708    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.291581 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.291549    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a8587d-openshift-default-dockercfg-zdp6b\""
Oct 31 18:33:02.326349 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.326322    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 18:33:02.336212 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336147    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.336323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336236    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.336323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336301    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.336403 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336348    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.336403 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336385    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.336519 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336497    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.336564 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336524    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.336564 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336558    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzmn\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-kube-api-access-qfzmn\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.336642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.336612    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.345218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.345187    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"]
Oct 31 18:33:02.437193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437162    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437198    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437325    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzmn\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-kube-api-access-qfzmn\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437365    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437444    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437527    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437583    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437637    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437695    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437709    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437712    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.437808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.437771    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.438109 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.438088    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.438309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.438292    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.439864 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.439841    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.440290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.440266    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.496182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.496091    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.497367 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.497335    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzmn\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-kube-api-access-qfzmn\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.553220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.553195    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:02.986593 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.986401    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"]
Oct 31 18:33:02.993420 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.993388    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:02.993537 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.993450    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:02.993537 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:02.993511    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:03.359836 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:03.359735    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" event={"ID":"1ce15580-1369-48c7-b20e-cec1a82cb6ed","Type":"ContainerStarted","Data":"7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7"}
Oct 31 18:33:03.359836 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:03.359802    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" event={"ID":"1ce15580-1369-48c7-b20e-cec1a82cb6ed","Type":"ContainerStarted","Data":"b9fde9fc0c4a374236b26c2eb8ecfa6bcd95520227b6422e9b8f9e2d0ac126d2"}
Oct 31 18:33:03.477269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:03.477207    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podStartSLOduration=1.477191108 podStartE2EDuration="1.477191108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:03.476131765 +0000 UTC m=+686.707370057" watchObservedRunningTime="2026-10-31 18:33:03.477191108 +0000 UTC m=+686.708429399"
Oct 31 18:33:03.553886 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:03.553847    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:04.559756 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:04.559725    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:05.372728 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:05.372685    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:05.373947 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:05.373922    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:15.207074 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.207029    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"]
Oct 31 18:33:15.208071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.208048    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d073d45-50fa-48a2-b95d-09d666a92991" containerName="istio-proxy"
Oct 31 18:33:15.213061 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.213038    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.269535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.269507    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-3e4a2c-serving-cert\""
Oct 31 18:33:15.276182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.276154    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"]
Oct 31 18:33:15.299156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.299130    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.299286 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.299168    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzg5v\" (UniqueName: \"kubernetes.io/projected/4429158d-d586-4a08-81bf-ba74734f82fc-kube-api-access-bzg5v\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.299286 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.299201    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-metrics-certs\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.299366 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.299317    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4429158d-d586-4a08-81bf-ba74734f82fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.400283 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.400251    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.400446 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.400292    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzg5v\" (UniqueName: \"kubernetes.io/projected/4429158d-d586-4a08-81bf-ba74734f82fc-kube-api-access-bzg5v\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.400446 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:33:15.400404    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-serving-cert: secret "maas-api-e2e-worker-w3-3e4a2c-serving-cert" not found
Oct 31 18:33:15.400446 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.400412    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-metrics-certs\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.400646 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:33:15.400490    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls podName:4429158d-d586-4a08-81bf-ba74734f82fc nodeName:}" failed. No retries permitted until 2026-10-31 18:33:15.900468064 +0000 UTC m=+699.131706347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls") pod "maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n" (UID: "4429158d-d586-4a08-81bf-ba74734f82fc") : secret "maas-api-e2e-worker-w3-3e4a2c-serving-cert" not found
Oct 31 18:33:15.400646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.400506    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4429158d-d586-4a08-81bf-ba74734f82fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.401159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.401137    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4429158d-d586-4a08-81bf-ba74734f82fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.403135 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.403111    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-metrics-certs\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.475823 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.475760    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzg5v\" (UniqueName: \"kubernetes.io/projected/4429158d-d586-4a08-81bf-ba74734f82fc-kube-api-access-bzg5v\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.906578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.906544    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:15.909160 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:15.909136    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:16.129013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:16.128983    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:16.533860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:16.533833    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"]
Oct 31 18:33:16.536196 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:33:16.536170    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4429158d_d586_4a08_81bf_ba74734f82fc.slice/crio-6cc2da63ff2be4ca612518bcb3fe423239d667699b72c919554439595b1a83b4 WatchSource:0}: Error finding container 6cc2da63ff2be4ca612518bcb3fe423239d667699b72c919554439595b1a83b4: Status 404 returned error can't find the container with id 6cc2da63ff2be4ca612518bcb3fe423239d667699b72c919554439595b1a83b4
Oct 31 18:33:16.605401 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:33:16.605370    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:33:17.447202 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:17.447168    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n" event={"ID":"4429158d-d586-4a08-81bf-ba74734f82fc","Type":"ContainerStarted","Data":"6cc2da63ff2be4ca612518bcb3fe423239d667699b72c919554439595b1a83b4"}
Oct 31 18:33:18.472493 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:18.471665    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n" event={"ID":"4429158d-d586-4a08-81bf-ba74734f82fc","Type":"ContainerStarted","Data":"4c409f2e39942ff013fcf709116aaa58f280c3dc6efdffce99c267739ae95ba5"}
Oct 31 18:33:18.473003 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:18.472654    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:18.587860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:18.587807    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n" podStartSLOduration=2.747485257 podStartE2EDuration="3.587791461s" totalImagesPullingTime="840.306204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:18.587485508 +0000 UTC m=+701.818723799" watchObservedRunningTime="2026-10-31 18:33:18.587791461 +0000 UTC m=+701.819029751"
Oct 31 18:33:23.521561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:23.520891    5551 generic.go:356] "Generic (PLEG): container finished" podID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerID="ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b" exitCode=0
Oct 31 18:33:23.521561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:23.521021    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" event={"ID":"9a0e9132-e0de-4ad8-bc9a-0e621973e643","Type":"ContainerDied","Data":"ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b"}
Oct 31 18:33:23.521561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:23.521542    5551 scope.go:122] "RemoveContainer" containerID="ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b"
Oct 31 18:33:23.523506 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:23.523445    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:23.523631 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:23.523573    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:23.523689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:23.523631    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:24.529254 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:24.529218    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" event={"ID":"9a0e9132-e0de-4ad8-bc9a-0e621973e643","Type":"ContainerStarted","Data":"a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5"}
Oct 31 18:33:25.038785 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:25.038756    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:33:25.039045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:25.038788    5551 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 18:33:25.039177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:25.039157    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 18:33:25.508489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:25.508095    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 18:33:26.038252 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:26.038221    5551 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 18:33:26.038786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:26.038273    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 18:33:27.038733 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:27.038698    5551 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 18:33:27.039224 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:27.038756    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 18:33:28.038746 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:28.038690    5551 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 18:33:28.039246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:28.038773    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 18:33:29.038885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:29.038797    5551 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 18:33:29.038885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:29.038872    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 18:33:30.042784 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:30.042749    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:33:30.043195 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:30.043159    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:33:30.044041 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:30.044019    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 18:33:32.602963 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:32.602865    5551 generic.go:356] "Generic (PLEG): container finished" podID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerID="c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d" exitCode=0
Oct 31 18:33:32.602963 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:32.602937    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" event={"ID":"c5ca5d99-8de5-497d-aae0-6e5a6536b739","Type":"ContainerDied","Data":"c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d"}
Oct 31 18:33:32.603487 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:32.603323    5551 scope.go:122] "RemoveContainer" containerID="c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d"
Oct 31 18:33:32.604765 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:32.604732    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:32.604859 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:32.604823    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:32.604903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:32.604867    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:33.610897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:33.610862    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" event={"ID":"c5ca5d99-8de5-497d-aae0-6e5a6536b739","Type":"ContainerStarted","Data":"012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b"}
Oct 31 18:33:34.387886 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:34.387849    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:33:34.389083 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:34.389055    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:33:34.389227 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:34.389111    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 18:33:35.388032 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:35.387995    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:33:35.388441 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:35.388058    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 18:33:36.388840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:36.388806    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:33:36.389252 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:36.388873    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 18:33:37.287417 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:37.286584    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:33:37.392481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:37.389253    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:33:37.392481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:37.389312    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 18:33:38.388926 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:38.388875    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:33:38.389120 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:38.388943    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 18:33:39.392937 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:39.392901    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:33:39.394114 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:39.394096    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:33:46.608521 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:33:46.608427    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:33:54.747737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:54.747703    5551 generic.go:356] "Generic (PLEG): container finished" podID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerID="7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7" exitCode=0
Oct 31 18:33:54.748170 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:54.747776    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" event={"ID":"1ce15580-1369-48c7-b20e-cec1a82cb6ed","Type":"ContainerDied","Data":"7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7"}
Oct 31 18:33:54.748170 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:54.748145    5551 scope.go:122] "RemoveContainer" containerID="7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7"
Oct 31 18:33:54.749387 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:54.749350    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:54.749523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:54.749446    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:54.749523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:54.749510    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:33:55.777357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:55.777317    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" event={"ID":"1ce15580-1369-48c7-b20e-cec1a82cb6ed","Type":"ContainerStarted","Data":"ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3"}
Oct 31 18:33:56.555483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:56.553789    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:33:56.555483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:56.553907    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:33:56.555483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:56.553962    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 18:33:57.555899 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:57.555862    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:33:57.556540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:57.556505    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 18:33:58.563273 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:58.562831    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:33:58.563273 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:58.562893    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 18:33:59.553869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:59.553829    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:33:59.554171 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:33:59.554139    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 18:34:00.554485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:00.553796    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:34:00.554485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:00.553867    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 18:34:01.559544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:01.559513    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:34:01.560040 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:01.559879    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:34:01.560567 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:01.560545    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 18:34:16.604401 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:34:16.604367    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:34:43.074844 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:43.074810    5551 generic.go:356] "Generic (PLEG): container finished" podID="2ff429fc-0746-4d6a-aff4-088caec1919e" containerID="d5b0a4c4983b4b1d4a5d6d1c9fab26676cfd5692eec0b97632e291893b42477f" exitCode=1
Oct 31 18:34:43.075403 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:43.074877    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn" event={"ID":"2ff429fc-0746-4d6a-aff4-088caec1919e","Type":"ContainerDied","Data":"d5b0a4c4983b4b1d4a5d6d1c9fab26676cfd5692eec0b97632e291893b42477f"}
Oct 31 18:34:43.075403 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:43.075244    5551 scope.go:122] "RemoveContainer" containerID="d5b0a4c4983b4b1d4a5d6d1c9fab26676cfd5692eec0b97632e291893b42477f"
Oct 31 18:34:44.083363 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:44.083328    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn" event={"ID":"2ff429fc-0746-4d6a-aff4-088caec1919e","Type":"ContainerStarted","Data":"7d52821541ea6a3fd920cd177badf98450876aadc654e7ec3b63e658603bc32c"}
Oct 31 18:34:44.083879 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:44.083687    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:34:46.603837 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:34:46.603803    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:34:55.090604 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:34:55.090575    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hd4jn"
Oct 31 18:35:16.605763 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:35:16.605673    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:35:46.604617 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:35:46.604575    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:35:49.750792 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:49.750758    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"]
Oct 31 18:35:49.751372 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:49.750970    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy" containerID="cri-o://05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8" gracePeriod=30
Oct 31 18:35:55.030775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.030707    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:35:55.067840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.067806    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istiod-ca-cert\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068039 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.067855    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-envoy\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068039 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.067896    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-podinfo\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068039 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.067977    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-kube-api-access-qf67v\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-kube-api-access-qf67v\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068039 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068014    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-token\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-token\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068048    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-credential-socket\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068093    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-socket\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068206    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istiod-ca-cert" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:35:55.068397 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068260    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-data\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068397 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068299    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-certs\") pod \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\" (UID: \"9525d3e3-170d-42d1-a7cf-89a87b80d0fc\") "
Oct 31 18:35:55.068606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068527    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-credential-socket" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.068606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068566    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-socket" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.068763 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068744    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.068820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068771    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.068820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068787    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.068820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068751    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-certs" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.069107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.068982    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-data" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.070851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.070824    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-podinfo" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:35:55.071161 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.071134    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-kube-api-access-qf67v" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "kube-api-access-qf67v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:35:55.071286 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.071270    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-token" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:35:55.073452 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.073425    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-envoy" pod "9525d3e3-170d-42d1-a7cf-89a87b80d0fc" (UID: "9525d3e3-170d-42d1-a7cf-89a87b80d0fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.170070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.170039    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.170070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.170065    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.170070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.170074    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.170347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.170084    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.170347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.170093    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf67v\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-kube-api-access-qf67v\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.170347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.170102    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9525d3e3-170d-42d1-a7cf-89a87b80d0fc-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.527649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.527615    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"]
Oct 31 18:35:55.528536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.528505    5551 cpu_manager.go:414] "containerMap: removing container" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy"
Oct 31 18:35:55.528536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.528526    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy"
Oct 31 18:35:55.528536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.528540    5551 cpu_manager.go:414] "containerMap: removing container" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy"
Oct 31 18:35:55.528730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.528545    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy"
Oct 31 18:35:55.528730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.528644    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy"
Oct 31 18:35:55.528730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.528652    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerName="istio-proxy"
Oct 31 18:35:55.532120 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.532104    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.555944 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.555910    5551 generic.go:356] "Generic (PLEG): container finished" podID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" containerID="05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8" exitCode=0
Oct 31 18:35:55.556081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.555963    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" event={"ID":"9525d3e3-170d-42d1-a7cf-89a87b80d0fc","Type":"ContainerDied","Data":"05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8"}
Oct 31 18:35:55.556081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.555971    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"
Oct 31 18:35:55.556081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.556050    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs" event={"ID":"9525d3e3-170d-42d1-a7cf-89a87b80d0fc","Type":"ContainerDied","Data":"69a0b7a0bb5eb059a9982f9e13a0a1e13e5a4594c53aa0f4648d7a30bb664424"}
Oct 31 18:35:55.556081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.556072    5551 scope.go:122] "RemoveContainer" containerID="05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8"
Oct 31 18:35:55.567897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.567873    5551 scope.go:122] "RemoveContainer" containerID="ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780"
Oct 31 18:35:55.573527 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573440    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b4dc00-057e-4d84-a25c-31ed75e4a187-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.573632 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573563    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.573632 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573619    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.573748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573651    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.573748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573691    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.573748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573723    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.573920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573799    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.573920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573830    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.573920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.573880    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txmfh\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-kube-api-access-txmfh\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.576800 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.576781    5551 scope.go:122] "RemoveContainer" containerID="05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8"
Oct 31 18:35:55.577087 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:35:55.577063    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8\": container with ID starting with 05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8 not found: ID does not exist" containerID="05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8"
Oct 31 18:35:55.577159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.577101    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8"} err="failed to get container status \"05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8\": rpc error: code = NotFound desc = could not find container \"05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8\": container with ID starting with 05651ac80ddd86960516663a1c8b5ae5eaecefa2d7490d3ff7be1d45e28e26f8 not found: ID does not exist"
Oct 31 18:35:55.577159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.577127    5551 scope.go:122] "RemoveContainer" containerID="ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780"
Oct 31 18:35:55.577427 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:35:55.577409    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780\": container with ID starting with ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780 not found: ID does not exist" containerID="ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780"
Oct 31 18:35:55.577515 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.577431    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780"} err="failed to get container status \"ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780\": rpc error: code = NotFound desc = could not find container \"ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780\": container with ID starting with ef371ddcf74a9711e6962ff0a3462511a12ae70f534510981a907b505142d780 not found: ID does not exist"
Oct 31 18:35:55.586880 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.586858    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-88085e-openshift-default-dockercfg-96c8d\""
Oct 31 18:35:55.638742 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.638710    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"]
Oct 31 18:35:55.675147 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675123    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b4dc00-057e-4d84-a25c-31ed75e4a187-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.675292 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675176    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.675292 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675227    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.675292 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675261    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.675451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675301    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.675451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675336    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.675451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675415    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.675649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675449    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.675649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675556    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txmfh\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-kube-api-access-txmfh\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.676067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675791    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.676067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.675947    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b4dc00-057e-4d84-a25c-31ed75e4a187-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.676067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.676033    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.676269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.676244    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.676413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.676395    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.677834 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.677816    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.677927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.677910    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.732051 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.732024    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.732263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.732247    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txmfh\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-kube-api-access-txmfh\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:55.762637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.762586    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"]
Oct 31 18:35:55.766815 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.766791    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rbrxs"]
Oct 31 18:35:55.846936 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:55.846861    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:56.279647 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.279607    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"]
Oct 31 18:35:56.281631 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:35:56.281600    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b4dc00_057e_4d84_a25c_31ed75e4a187.slice/crio-c33818ef4ddfdda324b36e7be1544251c33383dcfd3d8547ea0484febe2c12f6 WatchSource:0}: Error finding container c33818ef4ddfdda324b36e7be1544251c33383dcfd3d8547ea0484febe2c12f6: Status 404 returned error can't find the container with id c33818ef4ddfdda324b36e7be1544251c33383dcfd3d8547ea0484febe2c12f6
Oct 31 18:35:56.283586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.283544    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:35:56.283699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.283628    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:35:56.283699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.283674    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:35:56.566404 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.565380    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" event={"ID":"10b4dc00-057e-4d84-a25c-31ed75e4a187","Type":"ContainerStarted","Data":"7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227"}
Oct 31 18:35:56.566404 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.565450    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" event={"ID":"10b4dc00-057e-4d84-a25c-31ed75e4a187","Type":"ContainerStarted","Data":"c33818ef4ddfdda324b36e7be1544251c33383dcfd3d8547ea0484febe2c12f6"}
Oct 31 18:35:56.685838 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.685785    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" podStartSLOduration=1.685768322 podStartE2EDuration="1.685768322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:35:56.683431393 +0000 UTC m=+859.914669694" watchObservedRunningTime="2026-10-31 18:35:56.685768322 +0000 UTC m=+859.917006613"
Oct 31 18:35:56.847485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.847386    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:35:56.909137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:56.909108    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9525d3e3-170d-42d1-a7cf-89a87b80d0fc" path="/var/lib/kubelet/pods/9525d3e3-170d-42d1-a7cf-89a87b80d0fc/volumes"
Oct 31 18:35:57.848922 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:57.848891    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 18:35:57.849325 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:57.848956    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 18:35:58.847583 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:58.847546    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 18:35:58.847750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:58.847611    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 18:35:59.847512 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:59.847451    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 18:35:59.847904 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:35:59.847532    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 18:36:00.847413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:00.847380    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 18:36:00.847731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:00.847511    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 18:36:01.847584 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:01.847544    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 18:36:01.847774 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:01.847605    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 18:36:02.974445 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:02.974410    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:36:02.974928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:02.974848    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:36:02.975632 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:02.975606    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:36:16.610054 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:36:16.610014    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:36:37.074098 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:37.074047    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:36:37.084352 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:37.084329    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:36:45.077748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.077668    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"]
Oct 31 18:36:45.083828 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.083805    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.137864 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.137835    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"]
Oct 31 18:36:45.184375 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.184347    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 18:36:45.184375 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.184370    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 18:36:45.184375 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.184377    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3a8304947dff\""
Oct 31 18:36:45.184375 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.184361    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 18:36:45.184673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.184354    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ac4f996e253c\""
Oct 31 18:36:45.184673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.184405    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 18:36:45.252372 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.252323    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-routing-overlay-volume\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.252608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.252476    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.252608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.252525    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-plugins-config-volume\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.252732 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.252605    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77n7c\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-kube-api-access-77n7c\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.357531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.355105    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-plugins-config-volume\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.357531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.355168    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77n7c\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-kube-api-access-77n7c\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.357531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.355284    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-routing-overlay-volume\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.357531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.355406    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.359067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.359040    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-plugins-config-volume\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.375507 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.359813    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-routing-overlay-volume\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.376243 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.376218    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.414887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.414852    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77n7c\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-kube-api-access-77n7c\") pod \"payload-processing-external-model-6cdfd7fc9f-q5v9v\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") " pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:45.704484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:45.697481    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:46.154795 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:46.154766    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"]
Oct 31 18:36:46.156343 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:36:46.156314    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d78d781_b43b_47bd_ab28_fd5d840ec642.slice/crio-6ff7797b772cc7ef38709b067f6c42fd67be5916861abc8748d871e1114a91b5 WatchSource:0}: Error finding container 6ff7797b772cc7ef38709b067f6c42fd67be5916861abc8748d871e1114a91b5: Status 404 returned error can't find the container with id 6ff7797b772cc7ef38709b067f6c42fd67be5916861abc8748d871e1114a91b5
Oct 31 18:36:46.609171 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:36:46.609071    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:36:46.971037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:46.971001    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v" event={"ID":"7d78d781-b43b-47bd-ab28-fd5d840ec642","Type":"ContainerStarted","Data":"b6a2d5c0f37fe012c0aaf2d67bc3d05e8c2de3790fe85b4910a34636dd68649c"}
Oct 31 18:36:46.971234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:46.971070    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:36:46.971234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:46.971098    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v" event={"ID":"7d78d781-b43b-47bd-ab28-fd5d840ec642","Type":"ContainerStarted","Data":"6ff7797b772cc7ef38709b067f6c42fd67be5916861abc8748d871e1114a91b5"}
Oct 31 18:36:47.088495 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:47.088403    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v" podStartSLOduration=2.08838493 podStartE2EDuration="2.08838493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:47.08547765 +0000 UTC m=+910.316715939" watchObservedRunningTime="2026-10-31 18:36:47.08838493 +0000 UTC m=+910.319623222"
Oct 31 18:36:54.070695 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.070664    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"]
Oct 31 18:36:54.074556 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.074535    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.171404 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.171371    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tmp-dir\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.171575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.171425    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-home\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.171575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.171537    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tls-certs\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.171697 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.171579    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-dshm\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.171697 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.171615    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dp6t\" (UniqueName: \"kubernetes.io/projected/def5ef8b-df6f-499d-b9f8-33e86b6666ed-kube-api-access-9dp6t\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.171894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.171712    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-model-cache\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.176496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.176476    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"kube-root-ca.crt\""
Oct 31 18:36:54.176652 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.176614    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"e2e-conflict-a-1a3e1910-kserve-self-signed-certs\""
Oct 31 18:36:54.176652 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.176594    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"default-dockercfg-5cdvq\""
Oct 31 18:36:54.176652 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.176629    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"openshift-service-ca.crt\""
Oct 31 18:36:54.179149 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.179128    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"]
Oct 31 18:36:54.272407 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.272373    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tmp-dir\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.272608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.272436    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-home\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.272608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.272526    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tls-certs\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.272608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.272554    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-dshm\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.272608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.272577    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dp6t\" (UniqueName: \"kubernetes.io/projected/def5ef8b-df6f-499d-b9f8-33e86b6666ed-kube-api-access-9dp6t\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.272756 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.272712    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-model-cache\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.272873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.272849    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tmp-dir\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.272954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.272906    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-home\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.273080 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.273059    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-model-cache\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.275061 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.275039    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-dshm\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.275262 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.275246    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tls-certs\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.328814 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.328760    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dp6t\" (UniqueName: \"kubernetes.io/projected/def5ef8b-df6f-499d-b9f8-33e86b6666ed-kube-api-access-9dp6t\") pod \"e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.386976 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.386947    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:54.796955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:54.796928    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"]
Oct 31 18:36:54.798950 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:36:54.798924    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddef5ef8b_df6f_499d_b9f8_33e86b6666ed.slice/crio-b4362302a5d59d593c037a07b410b22a549299d846a916e63ae39a715af45361 WatchSource:0}: Error finding container b4362302a5d59d593c037a07b410b22a549299d846a916e63ae39a715af45361: Status 404 returned error can't find the container with id b4362302a5d59d593c037a07b410b22a549299d846a916e63ae39a715af45361
Oct 31 18:36:55.028982 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:55.028943    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx" event={"ID":"def5ef8b-df6f-499d-b9f8-33e86b6666ed","Type":"ContainerStarted","Data":"5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a"}
Oct 31 18:36:55.029165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:55.028991    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx" event={"ID":"def5ef8b-df6f-499d-b9f8-33e86b6666ed","Type":"ContainerStarted","Data":"b4362302a5d59d593c037a07b410b22a549299d846a916e63ae39a715af45361"}
Oct 31 18:36:55.141639 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:55.141575    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx" podStartSLOduration=1.141561958 podStartE2EDuration="1.141561958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:55.139381056 +0000 UTC m=+918.370619370" watchObservedRunningTime="2026-10-31 18:36:55.141561958 +0000 UTC m=+918.372800248"
Oct 31 18:36:55.387511 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:55.387450    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:36:57.981420 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:36:57.981392    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:37:04.387795 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:04.387747    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:37:04.404768 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:04.404747    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:37:05.108532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.108504    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:37:05.334344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.334312    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"]
Oct 31 18:37:05.338527 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.338506    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.352224 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.352194    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"]
Oct 31 18:37:05.357021 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.356992    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.397143 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.397108    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"]
Oct 31 18:37:05.446359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.446325    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"]
Oct 31 18:37:05.509794 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.509752    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.510088 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510055    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510109    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.510220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510145    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.510220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510164    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.510376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510223    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510252    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510272    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510302    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.510376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510327    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510349    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510377    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrnm\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-kube-api-access-kkrnm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510409    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510432    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.510659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510488    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26shq\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-kube-api-access-26shq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.510872 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510850    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.510909 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510892    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.510942 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.510918    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.612268 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612234    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.612525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612275    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.612525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612298    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrnm\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-kube-api-access-kkrnm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.612525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612327    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.612525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612353    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.612703 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612516    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26shq\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-kube-api-access-26shq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.612703 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612680    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.612772 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612732    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.612826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612769    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.612826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612775    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.612933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612828    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.612933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612840    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.612933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612887    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.612933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612929    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.613131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612961    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.613131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.612992    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.613131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613036    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.613131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613072    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.613131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613104    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.613131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613102    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.613430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613153    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.613430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613251    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.613430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613384    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.613430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613396    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.613667 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613420    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.613721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613693    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.613778 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.613749    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.614131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.614105    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.615284 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.615232    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.615664 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.615637    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.615764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.615697    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.615873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.615852    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.669875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.669835    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.670431 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.670401    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.670609 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.670497    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrnm\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-kube-api-access-kkrnm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zns87\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:05.670899 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.670880    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26shq\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-kube-api-access-26shq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dsbtv\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.672863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.672836    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:05.959052 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:05.959022    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:06.107159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:06.107130    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"]
Oct 31 18:37:06.111970 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:37:06.111943    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6edefc_ce6a_4e9f_80e7_e39bb4e68550.slice/crio-ce48068027cd9f51b71f16f0866be99d4d53782b0458aa6239b571d8aedc28e7 WatchSource:0}: Error finding container ce48068027cd9f51b71f16f0866be99d4d53782b0458aa6239b571d8aedc28e7: Status 404 returned error can't find the container with id ce48068027cd9f51b71f16f0866be99d4d53782b0458aa6239b571d8aedc28e7
Oct 31 18:37:06.114327 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:06.114286    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:06.114439 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:06.114362    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:06.114439 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:06.114409    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:06.339586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:06.339559    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"]
Oct 31 18:37:06.341853 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:37:06.341826    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e415b36_4318_42d4_a36c_ca99f48cfd7f.slice/crio-ec2be177738eb6ed83f32d6367910cdfcc872b71d9db08bb23c5ddd290477e6e WatchSource:0}: Error finding container ec2be177738eb6ed83f32d6367910cdfcc872b71d9db08bb23c5ddd290477e6e: Status 404 returned error can't find the container with id ec2be177738eb6ed83f32d6367910cdfcc872b71d9db08bb23c5ddd290477e6e
Oct 31 18:37:06.344610 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:06.344573    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:06.344686 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:06.344662    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:06.344733 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:06.344709    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:07.117479 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.116991    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" event={"ID":"de6edefc-ce6a-4e9f-80e7-e39bb4e68550","Type":"ContainerStarted","Data":"786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302"}
Oct 31 18:37:07.120612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.118732    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" event={"ID":"de6edefc-ce6a-4e9f-80e7-e39bb4e68550","Type":"ContainerStarted","Data":"ce48068027cd9f51b71f16f0866be99d4d53782b0458aa6239b571d8aedc28e7"}
Oct 31 18:37:07.120612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.119338    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" event={"ID":"2e415b36-4318-42d4-a36c-ca99f48cfd7f","Type":"ContainerStarted","Data":"c4773ab8d32d17d28a7039cd7f73d21753c64c44e74c30ca7022c895ed75f5ea"}
Oct 31 18:37:07.120612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.119374    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" event={"ID":"2e415b36-4318-42d4-a36c-ca99f48cfd7f","Type":"ContainerStarted","Data":"ec2be177738eb6ed83f32d6367910cdfcc872b71d9db08bb23c5ddd290477e6e"}
Oct 31 18:37:07.304483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.300857    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podStartSLOduration=2.300839098 podStartE2EDuration="2.300839098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:07.298322044 +0000 UTC m=+930.529560334" watchObservedRunningTime="2026-10-31 18:37:07.300839098 +0000 UTC m=+930.532077389"
Oct 31 18:37:07.440347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.437841    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podStartSLOduration=2.43780885 podStartE2EDuration="2.43780885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:07.433794972 +0000 UTC m=+930.665033262" watchObservedRunningTime="2026-10-31 18:37:07.43780885 +0000 UTC m=+930.669047141"
Oct 31 18:37:07.686483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.675246    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:07.686483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.675432    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 18:37:07.686483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.675501    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 18:37:07.962486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.962134    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:07.962486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.962271    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:37:07.962486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:07.962317    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 18:37:08.685946 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.685906    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 18:37:08.686597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.686564    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 18:37:08.804729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.804696    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"]
Oct 31 18:37:08.809172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.809150    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:08.920480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.917748    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4ptj5\""
Oct 31 18:37:08.924603 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.924306    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-provider-credentials\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:08.924603 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.924389    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-routing-overlay-volume\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:08.924603 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.924503    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-plugins-config-volume\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:08.924866 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.924639    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"]
Oct 31 18:37:08.924866 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.924755    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9fk\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-kube-api-access-2r9fk\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:08.962901 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.962826    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:37:08.963124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:08.963096    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 18:37:09.025659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.025619    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-provider-credentials\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.025935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.025919    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-routing-overlay-volume\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.026104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.026091    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-plugins-config-volume\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.026210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.026200    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9fk\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-kube-api-access-2r9fk\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.027164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.027143    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-routing-overlay-volume\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.027430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.027409    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-plugins-config-volume\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.028503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.028473    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-provider-credentials\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.102249 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.102216    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9fk\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-kube-api-access-2r9fk\") pod \"payload-processing-external-model-5c5b4d86b6-2sbwn\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") " pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.123851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.123817    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:09.584037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.583707    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"]
Oct 31 18:37:09.606141 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:37:09.606103    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1081b128_e2ba_4f64_8aea_335fc79d8b7c.slice/crio-62f76728b6905412f885ea9e6eb4b55a914573c707828a49bb835be131327bf0 WatchSource:0}: Error finding container 62f76728b6905412f885ea9e6eb4b55a914573c707828a49bb835be131327bf0: Status 404 returned error can't find the container with id 62f76728b6905412f885ea9e6eb4b55a914573c707828a49bb835be131327bf0
Oct 31 18:37:09.694680 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.691883    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 18:37:09.694680 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.691941    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 18:37:09.980492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.979559    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:37:09.980492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:09.979619    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 18:37:10.213270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:10.213211    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" event={"ID":"1081b128-e2ba-4f64-8aea-335fc79d8b7c","Type":"ContainerStarted","Data":"be03595494e591c4072037e16f7e0fc23cc1e4576a8639036746550edfcbc715"}
Oct 31 18:37:10.213450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:10.213290    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" event={"ID":"1081b128-e2ba-4f64-8aea-335fc79d8b7c","Type":"ContainerStarted","Data":"62f76728b6905412f885ea9e6eb4b55a914573c707828a49bb835be131327bf0"}
Oct 31 18:37:10.213569 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:10.213514    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:10.335290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:10.335174    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" podStartSLOduration=2.335159344 podStartE2EDuration="2.335159344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:10.333195021 +0000 UTC m=+933.564433322" watchObservedRunningTime="2026-10-31 18:37:10.335159344 +0000 UTC m=+933.566397630"
Oct 31 18:37:10.678498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:10.674845    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 18:37:10.678498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:10.674906    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 18:37:10.961543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:10.959869    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:37:10.961543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:10.959931    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 18:37:11.681483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:11.680861    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 18:37:11.681483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:11.680926    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 18:37:11.971540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:11.961741    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:37:11.971540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:11.961809    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 18:37:12.673852 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:12.673816    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 18:37:12.674031 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:12.673896    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 18:37:12.960370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:12.960258    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:37:12.960370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:12.960329    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 18:37:13.674297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:13.674246    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 18:37:13.674891 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:13.674303    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 18:37:13.959538 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:13.959442    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:37:13.959538 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:13.959525    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 18:37:14.914738 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:14.914697    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:14.915569 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:14.915544    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:14.916636 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:14.916614    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:37:15.151030 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:15.150992    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:15.151559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:15.151534    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:15.152508 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:15.152487    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:37:16.607916 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:37:16.607878    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:37:20.486424 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.486383    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"]
Oct 31 18:37:20.546583 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.546544    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.679484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.650276    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"]
Oct 31 18:37:20.679756 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.657878    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.679997 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.679975    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.683481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.680134    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.711481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.683644    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk7wq\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-kube-api-access-xk7wq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.711721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.711695    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.711846 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.711830    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.711994 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.711978    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.712110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.712095    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.712216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.712202    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.832548 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.823195    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.833306 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.833276    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.833547 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.833526    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.833731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.833693    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk7wq\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-kube-api-access-xk7wq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.833933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.833912    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.834088 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.834070    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.834279 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.834249    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.834421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.834404    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.834625 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.834574    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.834881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.834666    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.835038 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.834906    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.835473 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.835422    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.835655 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.835564    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.835840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.835800    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.840393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.839853    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.840393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.840320    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.917361 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.917311    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:20.946233 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:20.946189    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk7wq\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-kube-api-access-xk7wq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rzjwf\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:21.036881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:21.036578    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:21.227147 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:21.226696    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" podUID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:37:21.552438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:21.552408    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"]
Oct 31 18:37:21.553976 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:37:21.553951    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ac852a_0582_422c_bc43_6f0820bcdbb1.slice/crio-b97ee80229b57824349381b14014e4c4dc8562a56c08ac00f42ec068a373a44f WatchSource:0}: Error finding container b97ee80229b57824349381b14014e4c4dc8562a56c08ac00f42ec068a373a44f: Status 404 returned error can't find the container with id b97ee80229b57824349381b14014e4c4dc8562a56c08ac00f42ec068a373a44f
Oct 31 18:37:21.556354 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:21.556328    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:21.556423 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:21.556410    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:21.556481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:21.556445    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:37:22.324019 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:22.323926    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" event={"ID":"c8ac852a-0582-422c-bc43-6f0820bcdbb1","Type":"ContainerStarted","Data":"ec813721f13529db9af15bcc7eaa139ee724879a3e98619b5b745526f9e28ecb"}
Oct 31 18:37:22.324019 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:22.323982    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" event={"ID":"c8ac852a-0582-422c-bc43-6f0820bcdbb1","Type":"ContainerStarted","Data":"b97ee80229b57824349381b14014e4c4dc8562a56c08ac00f42ec068a373a44f"}
Oct 31 18:37:22.570486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:22.569976    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podStartSLOduration=2.5699496870000003 podStartE2EDuration="2.569949687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:22.499764733 +0000 UTC m=+945.731003035" watchObservedRunningTime="2026-10-31 18:37:22.569949687 +0000 UTC m=+945.801187978"
Oct 31 18:37:23.036838 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:23.036796    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:23.038699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:23.038667    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 18:37:23.038860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:23.038726    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 18:37:24.037988 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:24.037949    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 18:37:24.038377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:24.038009    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 18:37:25.037118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:25.037083    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 18:37:25.037404 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:25.037172    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 18:37:26.037752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:26.037723    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 18:37:26.038154 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:26.037801    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 18:37:27.037213 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:27.037176    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 18:37:27.037410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:27.037231    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 18:37:28.037967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:28.037920    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 18:37:28.038395 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:28.038015    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 18:37:29.037541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:29.037504    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 18:37:29.037719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:29.037558    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 18:37:30.042540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:30.042505    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:30.043951 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:30.043919    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:30.045870 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:30.044994    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:37:31.224271 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:31.224235    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" podUID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:37:40.956193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:40.956146    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"]
Oct 31 18:37:40.957550 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:40.957525    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" podUID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:37:40.989176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:40.989138    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-4px67"]
Oct 31 18:37:40.994254 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:40.994237    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.068182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.068152    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-4px67"]
Oct 31 18:37:41.094841 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.094794    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-plugins-config-volume\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.095061 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.094959    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snc7x\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-kube-api-access-snc7x\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.095061 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.095027    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-routing-overlay-volume\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.095174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.095151    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-provider-credentials\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.196749 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.196720    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-routing-overlay-volume\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.196906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.196781    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-provider-credentials\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.196906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.196881    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-plugins-config-volume\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.196979 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.196921    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snc7x\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-kube-api-access-snc7x\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.197368 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.197327    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-routing-overlay-volume\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.197676 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.197658    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-plugins-config-volume\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.199337 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.199317    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-provider-credentials\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.254745 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.254681    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snc7x\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-kube-api-access-snc7x\") pod \"payload-processing-external-model-f55c59c9-4px67\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.308481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.308431    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:41.689646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:41.689608    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-4px67"]
Oct 31 18:37:41.693360 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:37:41.692419    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ab1515_7b9b_4f0f_9c4a_5d7f4c9811dd.slice/crio-abe945fa652b9d8960e116eca4781b55bb02b4b769c31eaf61d62f9c6003b7db WatchSource:0}: Error finding container abe945fa652b9d8960e116eca4781b55bb02b4b769c31eaf61d62f9c6003b7db: Status 404 returned error can't find the container with id abe945fa652b9d8960e116eca4781b55bb02b4b769c31eaf61d62f9c6003b7db
Oct 31 18:37:42.475669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:42.475633    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" event={"ID":"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd","Type":"ContainerStarted","Data":"74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c"}
Oct 31 18:37:42.476174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:42.475718    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" event={"ID":"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd","Type":"ContainerStarted","Data":"abe945fa652b9d8960e116eca4781b55bb02b4b769c31eaf61d62f9c6003b7db"}
Oct 31 18:37:42.476174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:42.475772    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:37:42.594111 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:42.594043    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" podStartSLOduration=2.594021593 podStartE2EDuration="2.594021593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:42.589319164 +0000 UTC m=+965.820557467" watchObservedRunningTime="2026-10-31 18:37:42.594021593 +0000 UTC m=+965.825259886"
Oct 31 18:37:45.956494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:45.956405    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" podUID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" containerName="payload-processing" containerID="cri-o://be03595494e591c4072037e16f7e0fc23cc1e4576a8639036746550edfcbc715" gracePeriod=25
Oct 31 18:37:46.864541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:46.856009    5551 generic.go:356] "Generic (PLEG): container finished" podID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" containerID="be03595494e591c4072037e16f7e0fc23cc1e4576a8639036746550edfcbc715" exitCode=0
Oct 31 18:37:46.864940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:46.864908    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" event={"ID":"1081b128-e2ba-4f64-8aea-335fc79d8b7c","Type":"ContainerDied","Data":"be03595494e591c4072037e16f7e0fc23cc1e4576a8639036746550edfcbc715"}
Oct 31 18:37:46.946623 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:46.938185    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:47.049491 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:37:47.048523    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:37:47.157483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.154090    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-kube-api-access-2r9fk\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-kube-api-access-2r9fk\") pod \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") "
Oct 31 18:37:47.161582 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.161549    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-provider-credentials\") pod \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") "
Oct 31 18:37:47.167549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.167524    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-plugins-config-volume\") pod \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") "
Oct 31 18:37:47.172484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.167873    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-routing-overlay-volume\") pod \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\" (UID: \"1081b128-e2ba-4f64-8aea-335fc79d8b7c\") "
Oct 31 18:37:47.204184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.204146    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-plugins-config-volume" pod "1081b128-e2ba-4f64-8aea-335fc79d8b7c" (UID: "1081b128-e2ba-4f64-8aea-335fc79d8b7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:37:47.248480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.222967    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-routing-overlay-volume" pod "1081b128-e2ba-4f64-8aea-335fc79d8b7c" (UID: "1081b128-e2ba-4f64-8aea-335fc79d8b7c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:37:47.277399 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.277328    5551 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-routing-overlay-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:37:47.277399 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.277373    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1081b128-e2ba-4f64-8aea-335fc79d8b7c-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:37:47.290481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.278081    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-provider-credentials" pod "1081b128-e2ba-4f64-8aea-335fc79d8b7c" (UID: "1081b128-e2ba-4f64-8aea-335fc79d8b7c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:37:47.403501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.380614    5551 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-provider-credentials\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:37:47.403748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.398632    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-kube-api-access-2r9fk" pod "1081b128-e2ba-4f64-8aea-335fc79d8b7c" (UID: "1081b128-e2ba-4f64-8aea-335fc79d8b7c"). InnerVolumeSpecName "kube-api-access-2r9fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:37:47.513537 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.512522    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r9fk\" (UniqueName: \"kubernetes.io/projected/1081b128-e2ba-4f64-8aea-335fc79d8b7c-kube-api-access-2r9fk\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:37:47.898501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.897992    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn" event={"ID":"1081b128-e2ba-4f64-8aea-335fc79d8b7c","Type":"ContainerDied","Data":"62f76728b6905412f885ea9e6eb4b55a914573c707828a49bb835be131327bf0"}
Oct 31 18:37:47.898501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.898067    5551 scope.go:122] "RemoveContainer" containerID="be03595494e591c4072037e16f7e0fc23cc1e4576a8639036746550edfcbc715"
Oct 31 18:37:47.898501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:47.898234    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"
Oct 31 18:37:48.130826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:48.130790    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"]
Oct 31 18:37:48.194053 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:48.186605    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c5b4d86b6-2sbwn"]
Oct 31 18:37:48.907920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:48.907892    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" path="/var/lib/kubelet/pods/1081b128-e2ba-4f64-8aea-335fc79d8b7c/volumes"
Oct 31 18:37:53.485920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:37:53.485882    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:38:02.002389 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:02.002360    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3a8304947dff: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:02.002389 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:02.002386    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:02.003116 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:02.002445    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials podName:7d78d781-b43b-47bd-ab28-fd5d840ec642 nodeName:}" failed. No retries permitted until 2026-10-31 18:38:02.502430073 +0000 UTC m=+985.733668340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials") pod "payload-processing-external-model-6cdfd7fc9f-q5v9v" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642") : secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:02.511339 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:02.511301    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3a8304947dff: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:02.511339 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:02.511335    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:02.511630 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:02.511388    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials podName:7d78d781-b43b-47bd-ab28-fd5d840ec642 nodeName:}" failed. No retries permitted until 2026-10-31 18:38:03.511373156 +0000 UTC m=+986.742611425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials") pod "payload-processing-external-model-6cdfd7fc9f-q5v9v" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642") : secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:03.484522 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:03.484487    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:38:03.523955 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:03.523927    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3a8304947dff: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:03.524102 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:03.523960    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:03.524102 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:03.524015    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials podName:7d78d781-b43b-47bd-ab28-fd5d840ec642 nodeName:}" failed. No retries permitted until 2026-10-31 18:38:05.523998899 +0000 UTC m=+988.755237169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials") pod "payload-processing-external-model-6cdfd7fc9f-q5v9v" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642") : secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:05.547129 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:05.547095    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3a8304947dff: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:05.547129 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:05.547126    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:05.547616 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:05.547188    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials podName:7d78d781-b43b-47bd-ab28-fd5d840ec642 nodeName:}" failed. No retries permitted until 2026-10-31 18:38:09.547173857 +0000 UTC m=+992.778412126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials") pod "payload-processing-external-model-6cdfd7fc9f-q5v9v" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642") : secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:07.605135 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.605106    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"]
Oct 31 18:38:07.605721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.605703    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" containerName="payload-processing"
Oct 31 18:38:07.605771 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.605722    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" containerName="payload-processing"
Oct 31 18:38:07.605830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.605821    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1081b128-e2ba-4f64-8aea-335fc79d8b7c" containerName="payload-processing"
Oct 31 18:38:07.610493 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.610475    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.708241 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.708200    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"]
Oct 31 18:38:07.773160 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773116    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9f330e-d686-4475-8579-a0d779d0230c-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.773347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773183    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.773401 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773360    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.773450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773416    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.773543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773527    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.773603 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773588    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cn6t\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-kube-api-access-6cn6t\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.773649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773634    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.773704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773667    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.773796 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.773733    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9f330e-d686-4475-8579-a0d779d0230c-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875239    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875443 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875298    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875443 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875334    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875505    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cn6t\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-kube-api-access-6cn6t\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875558    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875582    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875762 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875625    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9f330e-d686-4475-8579-a0d779d0230c-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875762 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875697    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875762 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875713    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9f330e-d686-4475-8579-a0d779d0230c-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875762 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875715    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875952 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875768    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.875952 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875778    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.876066 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.875968    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.876281 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.876260    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9f330e-d686-4475-8579-a0d779d0230c-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.877929 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.877908    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.878216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.878198    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9f330e-d686-4475-8579-a0d779d0230c-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.932112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.932079    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:07.932272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:07.932253    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cn6t\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-kube-api-access-6cn6t\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:08.225412 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:08.225298    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:08.659748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:08.659716    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"]
Oct 31 18:38:08.661273 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:38:08.661246    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9f330e_d686_4475_8579_a0d779d0230c.slice/crio-e98bee01d1cb3fa0f5300d220a658e5dbdaecfba33bac2b3fc38ae49e07fb35f WatchSource:0}: Error finding container e98bee01d1cb3fa0f5300d220a658e5dbdaecfba33bac2b3fc38ae49e07fb35f: Status 404 returned error can't find the container with id e98bee01d1cb3fa0f5300d220a658e5dbdaecfba33bac2b3fc38ae49e07fb35f
Oct 31 18:38:08.663629 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:08.663599    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:38:08.663706 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:08.663657    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:38:08.663754 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:08.663714    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:38:09.037145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:09.036744    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" event={"ID":"5f9f330e-d686-4475-8579-a0d779d0230c","Type":"ContainerStarted","Data":"bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c"}
Oct 31 18:38:09.037145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:09.036809    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" event={"ID":"5f9f330e-d686-4475-8579-a0d779d0230c","Type":"ContainerStarted","Data":"e98bee01d1cb3fa0f5300d220a658e5dbdaecfba33bac2b3fc38ae49e07fb35f"}
Oct 31 18:38:09.226296 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:09.226261    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:09.227978 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:09.227955    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 18:38:09.228109 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:09.228001    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 18:38:09.597768 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:09.597733    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3a8304947dff: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:09.597768 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:09.597765    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:09.597984 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:09.597826    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials podName:7d78d781-b43b-47bd-ab28-fd5d840ec642 nodeName:}" failed. No retries permitted until 2026-10-31 18:38:17.597806859 +0000 UTC m=+1000.829045134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials") pod "payload-processing-external-model-6cdfd7fc9f-q5v9v" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642") : secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:10.226339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:10.226298    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 18:38:10.226853 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:10.226370    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 18:38:11.225953 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:11.225920    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 18:38:11.226124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:11.225968    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 18:38:12.226552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:12.226509    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 18:38:12.226958 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:12.226559    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 18:38:13.226152 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:13.226110    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 18:38:13.226488 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:13.226174    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 18:38:13.484807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:13.484720    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:38:14.226181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:14.226147    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 18:38:14.226349 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:14.226201    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 18:38:15.226415 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:15.226381    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 18:38:15.226844 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:15.226435    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 18:38:16.230567 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:16.230540    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:16.230950 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:16.230921    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:16.231608 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:16.231591    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:38:16.449536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:16.449445    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podStartSLOduration=9.449429463 podStartE2EDuration="9.449429463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:38:09.153837425 +0000 UTC m=+992.385075718" watchObservedRunningTime="2026-10-31 18:38:16.449429463 +0000 UTC m=+999.680667746"
Oct 31 18:38:16.606398 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:16.606319    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:38:17.616921 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:17.616894    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3a8304947dff: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:17.616921 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:17.616920    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:17.617336 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:17.616970    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials podName:7d78d781-b43b-47bd-ab28-fd5d840ec642 nodeName:}" failed. No retries permitted until 2026-10-31 18:38:33.616956572 +0000 UTC m=+1016.848194839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials") pod "payload-processing-external-model-6cdfd7fc9f-q5v9v" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642") : secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:21.145183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.145141    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-4px67"]
Oct 31 18:38:21.146532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.146498    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:38:21.177131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.177090    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"]
Oct 31 18:38:21.181171 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.181144    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.233770 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.233739    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ef93f66a26be\""
Oct 31 18:38:21.233955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.233823    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cb01f4a41ebc\""
Oct 31 18:38:21.237113 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.237082    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"]
Oct 31 18:38:21.266957 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.266920    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-plugins-config-volume\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.267145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.266977    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2sx4\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-kube-api-access-n2sx4\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.267145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.267024    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-routing-overlay-volume\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.267267 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.267181    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-provider-credentials\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.368376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.368333    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-routing-overlay-volume\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.368589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.368481    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-provider-credentials\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.368589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.368582    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-plugins-config-volume\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.368711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.368618    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2sx4\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-kube-api-access-n2sx4\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.369051 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.369032    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-routing-overlay-volume\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.369505 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.369481    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-plugins-config-volume\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.371303 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.371248    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-provider-credentials\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.425714 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.425637    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2sx4\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-kube-api-access-n2sx4\") pod \"payload-processing-external-model-7869f7dc8d-plpcb\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") " pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.496720 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.496689    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:21.871710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:21.871684    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"]
Oct 31 18:38:21.873658 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:38:21.873617    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9bef38_5973_4332_aa0a_d826734112ba.slice/crio-f63999e15509480995896f625c9dbff64038aa4dc36eee3892d8d8231e612a09 WatchSource:0}: Error finding container f63999e15509480995896f625c9dbff64038aa4dc36eee3892d8d8231e612a09: Status 404 returned error can't find the container with id f63999e15509480995896f625c9dbff64038aa4dc36eee3892d8d8231e612a09
Oct 31 18:38:22.148981 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:22.148952    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb" event={"ID":"0e9bef38-5973-4332-aa0a-d826734112ba","Type":"ContainerStarted","Data":"c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40"}
Oct 31 18:38:22.149407 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:22.149005    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb" event={"ID":"0e9bef38-5973-4332-aa0a-d826734112ba","Type":"ContainerStarted","Data":"f63999e15509480995896f625c9dbff64038aa4dc36eee3892d8d8231e612a09"}
Oct 31 18:38:22.149407 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:22.149143    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:22.263611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:22.263547    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb" podStartSLOduration=1.263530159 podStartE2EDuration="1.263530159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:38:22.262975957 +0000 UTC m=+1005.494214248" watchObservedRunningTime="2026-10-31 18:38:22.263530159 +0000 UTC m=+1005.494768451"
Oct 31 18:38:26.146243 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.146203    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerName="payload-processing" containerID="cri-o://74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c" gracePeriod=25
Oct 31 18:38:26.410254 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.410194    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:38:26.537784 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.537753    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-kube-api-access-snc7x\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-kube-api-access-snc7x\") pod \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") "
Oct 31 18:38:26.537964 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.537881    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-routing-overlay-volume\") pod \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") "
Oct 31 18:38:26.537964 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.537910    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-plugins-config-volume\") pod \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") "
Oct 31 18:38:26.538100 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.537994    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-provider-credentials\") pod \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\" (UID: \"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd\") "
Oct 31 18:38:26.538299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.538265    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-routing-overlay-volume" pod "80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" (UID: "80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:38:26.538415 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.538365    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-plugins-config-volume" pod "80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" (UID: "80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:38:26.538588 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.538573    5551 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-routing-overlay-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:38:26.538660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.538590    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:38:26.541592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.541567    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-kube-api-access-snc7x" pod "80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" (UID: "80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd"). InnerVolumeSpecName "kube-api-access-snc7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:38:26.542373 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.542351    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-provider-credentials" pod "80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" (UID: "80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:38:26.639769 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.639739    5551 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-provider-credentials\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:38:26.639769 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:26.639764    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snc7x\" (UniqueName: \"kubernetes.io/projected/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd-kube-api-access-snc7x\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:38:27.188949 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.188849    5551 generic.go:356] "Generic (PLEG): container finished" podID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerID="74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c" exitCode=0
Oct 31 18:38:27.188949 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.188893    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" event={"ID":"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd","Type":"ContainerDied","Data":"74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c"}
Oct 31 18:38:27.188949 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.188930    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67"
Oct 31 18:38:27.189551 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.188961    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f55c59c9-4px67" event={"ID":"80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd","Type":"ContainerDied","Data":"abe945fa652b9d8960e116eca4781b55bb02b4b769c31eaf61d62f9c6003b7db"}
Oct 31 18:38:27.189551 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.188990    5551 scope.go:122] "RemoveContainer" containerID="74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c"
Oct 31 18:38:27.200023 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.200004    5551 scope.go:122] "RemoveContainer" containerID="74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c"
Oct 31 18:38:27.200328 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:27.200309    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c\": container with ID starting with 74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c not found: ID does not exist" containerID="74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c"
Oct 31 18:38:27.200396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.200342    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c"} err="failed to get container status \"74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c\": rpc error: code = NotFound desc = could not find container \"74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c\": container with ID starting with 74607a3d5bc1521ce04b0fe39ed42006f0712503a382f805d57595fb6068115c not found: ID does not exist"
Oct 31 18:38:27.352678 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.352651    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-4px67"]
Oct 31 18:38:27.402389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:27.402361    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-4px67"]
Oct 31 18:38:28.908899 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:28.908867    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" path="/var/lib/kubelet/pods/80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd/volumes"
Oct 31 18:38:33.165845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:33.165816    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:38:33.333038 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:33.333001    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"]
Oct 31 18:38:33.618806 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:33.618712    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3a8304947dff: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:33.619014 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:33.618994    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v: secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:33.619191 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:33.619177    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials podName:7d78d781-b43b-47bd-ab28-fd5d840ec642 nodeName:}" failed. No retries permitted until 2026-10-31 18:39:05.619154839 +0000 UTC m=+1048.850393122 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials") pod "payload-processing-external-model-6cdfd7fc9f-q5v9v" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642") : secret "agc-cred-3a8304947dff" not found
Oct 31 18:38:37.162486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:37.161638    5551 scope.go:122] "RemoveContainer" containerID="3f447a0ad1b7a01a481489eb63abafe25f9a0fc9e8b9db112e8468c47d42f5b5"
Oct 31 18:38:37.187108 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:37.185308    5551 scope.go:122] "RemoveContainer" containerID="e490350868a81cb9eda6438a3b8482bd55baeaadaf13052f2fad8643c74bd685"
Oct 31 18:38:38.346485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:38.346101    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v" podUID="7d78d781-b43b-47bd-ab28-fd5d840ec642" containerName="payload-processing" containerID="cri-o://b6a2d5c0f37fe012c0aaf2d67bc3d05e8c2de3790fe85b4910a34636dd68649c" gracePeriod=25
Oct 31 18:38:40.369919 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.369881    5551 generic.go:356] "Generic (PLEG): container finished" podID="7d78d781-b43b-47bd-ab28-fd5d840ec642" containerID="b6a2d5c0f37fe012c0aaf2d67bc3d05e8c2de3790fe85b4910a34636dd68649c" exitCode=0
Oct 31 18:38:40.370372 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.369973    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v" event={"ID":"7d78d781-b43b-47bd-ab28-fd5d840ec642","Type":"ContainerDied","Data":"b6a2d5c0f37fe012c0aaf2d67bc3d05e8c2de3790fe85b4910a34636dd68649c"}
Oct 31 18:38:40.528731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.528663    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:38:40.630918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.630882    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials\") pod \"7d78d781-b43b-47bd-ab28-fd5d840ec642\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") "
Oct 31 18:38:40.631107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.630947    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-routing-overlay-volume\") pod \"7d78d781-b43b-47bd-ab28-fd5d840ec642\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") "
Oct 31 18:38:40.631107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.630973    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-plugins-config-volume\") pod \"7d78d781-b43b-47bd-ab28-fd5d840ec642\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") "
Oct 31 18:38:40.631218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.631150    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-kube-api-access-77n7c\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-kube-api-access-77n7c\") pod \"7d78d781-b43b-47bd-ab28-fd5d840ec642\" (UID: \"7d78d781-b43b-47bd-ab28-fd5d840ec642\") "
Oct 31 18:38:40.631366 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.631331    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-routing-overlay-volume" pod "7d78d781-b43b-47bd-ab28-fd5d840ec642" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:38:40.631474 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.631386    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-plugins-config-volume" pod "7d78d781-b43b-47bd-ab28-fd5d840ec642" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:38:40.631753 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.631729    5551 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-routing-overlay-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:38:40.631832 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.631758    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d78d781-b43b-47bd-ab28-fd5d840ec642-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:38:40.633207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.633184    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials" pod "7d78d781-b43b-47bd-ab28-fd5d840ec642" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:38:40.633433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.633415    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-kube-api-access-77n7c" pod "7d78d781-b43b-47bd-ab28-fd5d840ec642" (UID: "7d78d781-b43b-47bd-ab28-fd5d840ec642"). InnerVolumeSpecName "kube-api-access-77n7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:38:40.732648 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.732613    5551 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-provider-credentials\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:38:40.732648 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:40.732641    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77n7c\" (UniqueName: \"kubernetes.io/projected/7d78d781-b43b-47bd-ab28-fd5d840ec642-kube-api-access-77n7c\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:38:41.377423 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.377391    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"
Oct 31 18:38:41.377423 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.377409    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v" event={"ID":"7d78d781-b43b-47bd-ab28-fd5d840ec642","Type":"ContainerDied","Data":"6ff7797b772cc7ef38709b067f6c42fd67be5916861abc8748d871e1114a91b5"}
Oct 31 18:38:41.377998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.377505    5551 scope.go:122] "RemoveContainer" containerID="b6a2d5c0f37fe012c0aaf2d67bc3d05e8c2de3790fe85b4910a34636dd68649c"
Oct 31 18:38:41.541141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.541109    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"]
Oct 31 18:38:41.590694 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.590667    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdfd7fc9f-q5v9v"]
Oct 31 18:38:41.834100 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.834018    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"]
Oct 31 18:38:41.834600 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.834586    5551 cpu_manager.go:414] "containerMap: removing container" podUID="7d78d781-b43b-47bd-ab28-fd5d840ec642" containerName="payload-processing"
Oct 31 18:38:41.834654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.834601    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d78d781-b43b-47bd-ab28-fd5d840ec642" containerName="payload-processing"
Oct 31 18:38:41.834654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.834638    5551 cpu_manager.go:414] "containerMap: removing container" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerName="payload-processing"
Oct 31 18:38:41.834654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.834643    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerName="payload-processing"
Oct 31 18:38:41.834750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.834721    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d78d781-b43b-47bd-ab28-fd5d840ec642" containerName="payload-processing"
Oct 31 18:38:41.834750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.834735    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ab1515-7b9b-4f0f-9c4a-5d7f4c9811dd" containerName="payload-processing"
Oct 31 18:38:41.839922 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.839900    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.896588 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.896557    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"]
Oct 31 18:38:41.947927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.947879    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.948081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.948011    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.948081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.948044    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355150cb-d1ce-4ff4-8c0c-d490ff467821-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.948197 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.948178    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.948307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.948292    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.948347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.948336    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.948389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.948375    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.948430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.948411    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:41.948503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:41.948436    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9bw\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-kube-api-access-gq9bw\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050132    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050190    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050208    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355150cb-d1ce-4ff4-8c0c-d490ff467821-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050274    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050382    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050421    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050480    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050513    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050740 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050540    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9bw\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-kube-api-access-gq9bw\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050740 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050615    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050740 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050670    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050883 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050865    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.050939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.050894    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.051264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.051008    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355150cb-d1ce-4ff4-8c0c-d490ff467821-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.053283 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.053254    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.053503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.053478    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.107339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.107270    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.107579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.107555    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9bw\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-kube-api-access-gq9bw\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.153250 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.153219    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:42.584358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.584333    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"]
Oct 31 18:38:42.585622 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:38:42.585596    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355150cb_d1ce_4ff4_8c0c_d490ff467821.slice/crio-b67014a37704b0c8f09ea791e4546bbf94ca45e33c322d137f14448c37a2e3d5 WatchSource:0}: Error finding container b67014a37704b0c8f09ea791e4546bbf94ca45e33c322d137f14448c37a2e3d5: Status 404 returned error can't find the container with id b67014a37704b0c8f09ea791e4546bbf94ca45e33c322d137f14448c37a2e3d5
Oct 31 18:38:42.588085 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.588052    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:38:42.588180 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.588120    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:38:42.588180 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.588151    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:38:42.909831 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:42.909796    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d78d781-b43b-47bd-ab28-fd5d840ec642" path="/var/lib/kubelet/pods/7d78d781-b43b-47bd-ab28-fd5d840ec642/volumes"
Oct 31 18:38:43.394563 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:43.394524    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" event={"ID":"355150cb-d1ce-4ff4-8c0c-d490ff467821","Type":"ContainerStarted","Data":"49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57"}
Oct 31 18:38:43.394766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:43.394593    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" event={"ID":"355150cb-d1ce-4ff4-8c0c-d490ff467821","Type":"ContainerStarted","Data":"b67014a37704b0c8f09ea791e4546bbf94ca45e33c322d137f14448c37a2e3d5"}
Oct 31 18:38:43.614071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:43.614019    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podStartSLOduration=2.61399946 podStartE2EDuration="2.61399946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:38:43.566946559 +0000 UTC m=+1026.798184869" watchObservedRunningTime="2026-10-31 18:38:43.61399946 +0000 UTC m=+1026.845237748"
Oct 31 18:38:44.161755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:44.161725    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:44.169579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:44.169550    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 18:38:44.169790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:44.169764    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 18:38:45.157792 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:45.157756    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 18:38:45.158506 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:45.158450    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 18:38:46.157485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:46.155745    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 18:38:46.157485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:46.155802    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 18:38:46.604188 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:38:46.604099    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:38:47.154497 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:47.153829    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 18:38:47.154497 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:47.153901    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 18:38:48.161483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:48.155815    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 18:38:48.161483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:48.155877    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 18:38:49.153766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:49.153728    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 18:38:49.153989 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:49.153803    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 18:38:50.159487 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:50.159442    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:50.159948 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:50.159815    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:50.160590 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:50.160571    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:38:58.859443 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.859407    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"]
Oct 31 18:38:58.863479 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.863445    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:58.916541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.916419    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"kube-root-ca.crt\""
Oct 31 18:38:58.946551 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.946529    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"openshift-service-ca.crt\""
Oct 31 18:38:58.966153 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.966129    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"default-dockercfg-dmdhf\""
Oct 31 18:38:58.966300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.966240    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"distinct-2-w0-a8587d-kserve-self-signed-certs\""
Oct 31 18:38:58.970807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.970780    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"]
Oct 31 18:38:58.977868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.977843    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-model-cache\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:58.977999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.977877    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017d9396-474b-4678-b129-ca8e8b4e8643-tls-certs\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:58.977999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.977968    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-home\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:58.978110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.977997    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlclc\" (UniqueName: \"kubernetes.io/projected/017d9396-474b-4678-b129-ca8e8b4e8643-kube-api-access-nlclc\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:58.978110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.978078    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-dshm\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:58.978218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:58.978168    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-tmp-dir\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.078951 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.078918    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-model-cache\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.078951 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.078959    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017d9396-474b-4678-b129-ca8e8b4e8643-tls-certs\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.079217 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.079084    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-home\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.079217 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.079144    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlclc\" (UniqueName: \"kubernetes.io/projected/017d9396-474b-4678-b129-ca8e8b4e8643-kube-api-access-nlclc\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.079217 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.079192    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-dshm\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.079394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.079279    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-tmp-dir\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.079485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.079389    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-model-cache\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.079711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.079687    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-tmp-dir\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.079972 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.079788    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-home\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.081628 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.081606    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-dshm\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.082133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.082112    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017d9396-474b-4678-b129-ca8e8b4e8643-tls-certs\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.136789 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.136759    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlclc\" (UniqueName: \"kubernetes.io/projected/017d9396-474b-4678-b129-ca8e8b4e8643-kube-api-access-nlclc\") pod \"distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") " pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.177810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.177784    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:38:59.595510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:38:59.595479    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"]
Oct 31 18:38:59.597111 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:38:59.597080    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod017d9396_474b_4678_b129_ca8e8b4e8643.slice/crio-5923add63ae59bbf28de4a4177cc8d570059c0ad83fa5a538ae65792e72a54b2 WatchSource:0}: Error finding container 5923add63ae59bbf28de4a4177cc8d570059c0ad83fa5a538ae65792e72a54b2: Status 404 returned error can't find the container with id 5923add63ae59bbf28de4a4177cc8d570059c0ad83fa5a538ae65792e72a54b2
Oct 31 18:39:00.528723 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:00.528686    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" event={"ID":"017d9396-474b-4678-b129-ca8e8b4e8643","Type":"ContainerStarted","Data":"14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b"}
Oct 31 18:39:00.529103 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:00.528733    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" event={"ID":"017d9396-474b-4678-b129-ca8e8b4e8643","Type":"ContainerStarted","Data":"5923add63ae59bbf28de4a4177cc8d570059c0ad83fa5a538ae65792e72a54b2"}
Oct 31 18:39:00.643874 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:00.643817    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podStartSLOduration=2.643799961 podStartE2EDuration="2.643799961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:39:00.642451511 +0000 UTC m=+1043.873689800" watchObservedRunningTime="2026-10-31 18:39:00.643799961 +0000 UTC m=+1043.875038251"
Oct 31 18:39:01.178549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:01.178518    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:39:09.178015 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:09.177968    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:39:09.192599 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:09.192571    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:39:09.606389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:09.606305    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:39:16.607393 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:39:16.607346    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:39:23.196897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.196867    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"]
Oct 31 18:39:23.200743 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.200675    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.274487 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274414    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.274662 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274495    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.274662 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274587    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.274662 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274640    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.274769 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274754    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.274807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274789    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.274843 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274820    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlcx\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-kube-api-access-ljlcx\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.274886 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274870    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.275009 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.274994    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.296702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.296665    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"]
Oct 31 18:39:23.376869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.376831    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.376959    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.376993    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377021    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlcx\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-kube-api-access-ljlcx\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377066    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377172    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377225    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377255    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377286    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377308    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377675 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377638    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377786    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.377988 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.377951    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.378060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.378008    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.380140 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.380114    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.380269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.380246    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.434121 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.434085    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.434296 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.434278    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlcx\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-kube-api-access-ljlcx\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.515300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.515203    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:23.948655 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.948626    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"]
Oct 31 18:39:23.950236 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:39:23.950207    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1a1aa0_84fb_4608_9b4f_f3ac3a1b304e.slice/crio-f1f871911f9f9830c2ad08827e23d981627498c03e0849baf04bf8b30451c06c WatchSource:0}: Error finding container f1f871911f9f9830c2ad08827e23d981627498c03e0849baf04bf8b30451c06c: Status 404 returned error can't find the container with id f1f871911f9f9830c2ad08827e23d981627498c03e0849baf04bf8b30451c06c
Oct 31 18:39:23.952807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.952776    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:39:23.952898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.952838    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:39:23.952898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:23.952873    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:39:24.724871 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:24.724825    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" event={"ID":"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e","Type":"ContainerStarted","Data":"644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92"}
Oct 31 18:39:24.725263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:24.724928    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" event={"ID":"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e","Type":"ContainerStarted","Data":"f1f871911f9f9830c2ad08827e23d981627498c03e0849baf04bf8b30451c06c"}
Oct 31 18:39:24.943603 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:24.943555    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" podStartSLOduration=1.9435408870000002 podStartE2EDuration="1.943540887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:39:24.892723299 +0000 UTC m=+1068.123961590" watchObservedRunningTime="2026-10-31 18:39:24.943540887 +0000 UTC m=+1068.174779176"
Oct 31 18:39:25.516296 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:25.516259    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:25.517956 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:25.517930    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:39:25.518078 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:25.517982    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 18:39:26.516000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:26.515965    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:39:26.516422 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:26.516020    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 18:39:27.516100 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:27.516069    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:39:27.516546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:27.516136    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 18:39:28.516112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:28.516077    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:39:28.516503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:28.516144    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 18:39:29.515855 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:29.515813    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:39:29.516055 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:29.515906    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 18:39:30.516042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:30.516006    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:39:30.516424 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:30.516064    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 18:39:31.520301 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:31.520265    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:31.520763 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:31.520717    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:31.521396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:31.521378    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:39:46.604341 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:39:46.604262    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:39:47.017274 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:39:47.017243    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"]
Oct 31 18:40:02.038377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.038332    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx" podUID="def5ef8b-df6f-499d-b9f8-33e86b6666ed" containerName="main" containerID="cri-o://5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a" gracePeriod=45
Oct 31 18:40:02.097927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.097885    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx" podUID="def5ef8b-df6f-499d-b9f8-33e86b6666ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 31 18:40:02.322689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.322656    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:40:02.417794 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.417766    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-home\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-home\") pod \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") "
Oct 31 18:40:02.417977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.417832    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def5ef8b-df6f-499d-b9f8-33e86b6666ed-kube-api-access-9dp6t\" (UniqueName: \"kubernetes.io/projected/def5ef8b-df6f-499d-b9f8-33e86b6666ed-kube-api-access-9dp6t\") pod \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") "
Oct 31 18:40:02.417977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.417857    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tls-certs\") pod \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") "
Oct 31 18:40:02.417977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.417878    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-dshm\") pod \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") "
Oct 31 18:40:02.417977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.417961    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-model-cache\") pod \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") "
Oct 31 18:40:02.418181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.418043    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tmp-dir\") pod \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\" (UID: \"def5ef8b-df6f-499d-b9f8-33e86b6666ed\") "
Oct 31 18:40:02.418181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.418069    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-home" pod "def5ef8b-df6f-499d-b9f8-33e86b6666ed" (UID: "def5ef8b-df6f-499d-b9f8-33e86b6666ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:40:02.418354 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.418321    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-model-cache" pod "def5ef8b-df6f-499d-b9f8-33e86b6666ed" (UID: "def5ef8b-df6f-499d-b9f8-33e86b6666ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:40:02.418354 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.418332    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tmp-dir" pod "def5ef8b-df6f-499d-b9f8-33e86b6666ed" (UID: "def5ef8b-df6f-499d-b9f8-33e86b6666ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:40:02.418589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.418569    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:02.418639 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.418597    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:02.418639 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.418612    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:02.420340 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.420313    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-dshm" pod "def5ef8b-df6f-499d-b9f8-33e86b6666ed" (UID: "def5ef8b-df6f-499d-b9f8-33e86b6666ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:40:02.420340 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.420330    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def5ef8b-df6f-499d-b9f8-33e86b6666ed-kube-api-access-9dp6t" pod "def5ef8b-df6f-499d-b9f8-33e86b6666ed" (UID: "def5ef8b-df6f-499d-b9f8-33e86b6666ed"). InnerVolumeSpecName "kube-api-access-9dp6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:40:02.420506 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.420359    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tls-certs" pod "def5ef8b-df6f-499d-b9f8-33e86b6666ed" (UID: "def5ef8b-df6f-499d-b9f8-33e86b6666ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:40:02.519357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.519318    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dp6t\" (UniqueName: \"kubernetes.io/projected/def5ef8b-df6f-499d-b9f8-33e86b6666ed-kube-api-access-9dp6t\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:02.519357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.519353    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def5ef8b-df6f-499d-b9f8-33e86b6666ed-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:02.519357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:02.519365    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5ef8b-df6f-499d-b9f8-33e86b6666ed-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:03.072399 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.072361    5551 generic.go:356] "Generic (PLEG): container finished" podID="def5ef8b-df6f-499d-b9f8-33e86b6666ed" containerID="5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a" exitCode=0
Oct 31 18:40:03.072399 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.072396    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx" event={"ID":"def5ef8b-df6f-499d-b9f8-33e86b6666ed","Type":"ContainerDied","Data":"5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a"}
Oct 31 18:40:03.072955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.072411    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"
Oct 31 18:40:03.072955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.072442    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx" event={"ID":"def5ef8b-df6f-499d-b9f8-33e86b6666ed","Type":"ContainerDied","Data":"b4362302a5d59d593c037a07b410b22a549299d846a916e63ae39a715af45361"}
Oct 31 18:40:03.072955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.072491    5551 scope.go:122] "RemoveContainer" containerID="5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a"
Oct 31 18:40:03.082639 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.082605    5551 scope.go:122] "RemoveContainer" containerID="5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a"
Oct 31 18:40:03.082890 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:40:03.082866    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a\": container with ID starting with 5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a not found: ID does not exist" containerID="5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a"
Oct 31 18:40:03.082963 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.082902    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a"} err="failed to get container status \"5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a\": rpc error: code = NotFound desc = could not find container \"5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a\": container with ID starting with 5743c3d595cc1ddf9b3ed0d447eb51d3e33759f2d56822c55b7cedf860f91a6a not found: ID does not exist"
Oct 31 18:40:03.236078 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.236005    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"]
Oct 31 18:40:03.241004 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:03.240976    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-a-1a3e1910-kserve-78d458984d-gz8vx"]
Oct 31 18:40:04.909117 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:04.909086    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def5ef8b-df6f-499d-b9f8-33e86b6666ed" path="/var/lib/kubelet/pods/def5ef8b-df6f-499d-b9f8-33e86b6666ed/volumes"
Oct 31 18:40:16.652652 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:40:16.652591    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:40:20.651869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.651833    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"]
Oct 31 18:40:20.652413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.652396    5551 cpu_manager.go:414] "containerMap: removing container" podUID="def5ef8b-df6f-499d-b9f8-33e86b6666ed" containerName="main"
Oct 31 18:40:20.652533 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.652414    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def5ef8b-df6f-499d-b9f8-33e86b6666ed" containerName="main"
Oct 31 18:40:20.652598 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.652535    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="def5ef8b-df6f-499d-b9f8-33e86b6666ed" containerName="main"
Oct 31 18:40:20.655931 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.655911    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.757313 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.757275    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"]
Oct 31 18:40:20.759605 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.759576    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.759741 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.759716    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbqm\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-kube-api-access-xrbqm\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.759819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.759794    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.759893 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.759882    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.759956 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.759908    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.759956 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.759942    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.760071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.759978    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.760071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.760004    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.760205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.760105    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.861734 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.861699    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.861905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.861788    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.861905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.861813    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.861905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.861841    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.861905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.861871    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.861905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.861891    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.862166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.861986    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.862166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.862043    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.862166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.862110    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrbqm\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-kube-api-access-xrbqm\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.862166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.862139    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.862359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.862149    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.862359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.862244    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.862476 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.862435    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.862705 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.862674    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.864366 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.864345    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.864634 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.864616    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.918656 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.918585    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.918656 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.918621    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrbqm\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-kube-api-access-xrbqm\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:20.968955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:20.968931    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:21.397137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:21.397107    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"]
Oct 31 18:40:21.398410 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:40:21.398382    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbe7fbe_5aff_41b8_92f2_bef6bf61e7da.slice/crio-524d928578c46acf130c5fa1fa388b3b06c082612f5d30ca8da032b6dccb3b1e WatchSource:0}: Error finding container 524d928578c46acf130c5fa1fa388b3b06c082612f5d30ca8da032b6dccb3b1e: Status 404 returned error can't find the container with id 524d928578c46acf130c5fa1fa388b3b06c082612f5d30ca8da032b6dccb3b1e
Oct 31 18:40:21.400769 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:21.400739    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:40:21.400828 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:21.400797    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:40:21.400867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:21.400830    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:40:22.279553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:22.279422    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" event={"ID":"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da","Type":"ContainerStarted","Data":"f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168"}
Oct 31 18:40:22.279553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:22.279515    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" event={"ID":"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da","Type":"ContainerStarted","Data":"524d928578c46acf130c5fa1fa388b3b06c082612f5d30ca8da032b6dccb3b1e"}
Oct 31 18:40:22.399924 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:22.399867    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podStartSLOduration=2.399854164 podStartE2EDuration="2.399854164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:22.397772078 +0000 UTC m=+1125.629010368" watchObservedRunningTime="2026-10-31 18:40:22.399854164 +0000 UTC m=+1125.631092530"
Oct 31 18:40:22.972488 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:22.971879    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:22.976575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:22.976536    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 18:40:22.976736 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:22.976606    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 18:40:23.975096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:23.974612    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 18:40:23.975096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:23.974694    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 18:40:24.975070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:24.974859    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 18:40:24.975070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:24.974922    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 18:40:25.972485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:25.971798    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 18:40:25.972485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:25.971863    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 18:40:26.980942 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:26.980903    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 18:40:26.981573 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:26.981537    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 18:40:27.969512 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:27.969443    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 18:40:27.969683 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:27.969540    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 18:40:28.974414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:28.974383    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:28.974894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:28.974811    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:28.975606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:28.975586    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:40:32.306603 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:32.306566    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:40:32.307183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:32.307151    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:40:32.583263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:32.583156    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:40:32.583430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:32.583256    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:40:38.776471 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.776429    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"]
Oct 31 18:40:38.782021 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.781992    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.846830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.846793    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"]
Oct 31 18:40:38.920955 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.920915    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.921184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.921055    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.921184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.921085    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.921184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.921121    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-kube-api-access-pkjv7\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.921344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.921251    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.921344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.921305    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa63e23-0c15-440c-9e30-3bb576e312f1-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.921344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.921330    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.921529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.921421    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:38.921529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:38.921451    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.023687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023411    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.023687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023503    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.023687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023580    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.023687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023686    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.023687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023710    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.023687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023758    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-kube-api-access-pkjv7\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.023687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023824    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.023687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023895    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa63e23-0c15-440c-9e30-3bb576e312f1-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.029996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.023940    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.029996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.024401    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.029996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.027744    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.029996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.028025    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.029996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.028076    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.029996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.028269    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.029996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.028757    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa63e23-0c15-440c-9e30-3bb576e312f1-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.035575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.035552    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.082006 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.081980    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.083799 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.083774    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-kube-api-access-pkjv7\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.096171 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.096149    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:39.531832 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.531799    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"]
Oct 31 18:40:39.571038 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:40:39.571013    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa63e23_0c15_440c_9e30_3bb576e312f1.slice/crio-bbc6bc4ac36fda17d35c93979cebf466ec87eb10f44e8f8f6aa9c215aff7ba5c WatchSource:0}: Error finding container bbc6bc4ac36fda17d35c93979cebf466ec87eb10f44e8f8f6aa9c215aff7ba5c: Status 404 returned error can't find the container with id bbc6bc4ac36fda17d35c93979cebf466ec87eb10f44e8f8f6aa9c215aff7ba5c
Oct 31 18:40:39.573401 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.573366    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:40:39.573525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.573449    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:40:39.573525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:39.573512    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:40:40.385964 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.385928    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"]
Oct 31 18:40:40.390293 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.390270    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.494761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.494726    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"kube-root-ca.crt\""
Oct 31 18:40:40.495018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.494814    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"openshift-service-ca.crt\""
Oct 31 18:40:40.495018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.494977    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"distinct-w2-88085e-kserve-self-signed-certs\""
Oct 31 18:40:40.495164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.495140    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"default-dockercfg-5cdvq\""
Oct 31 18:40:40.498162 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.498137    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"]
Oct 31 18:40:40.505938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.505905    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" event={"ID":"aaa63e23-0c15-440c-9e30-3bb576e312f1","Type":"ContainerStarted","Data":"bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb"}
Oct 31 18:40:40.506061 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.505965    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" event={"ID":"aaa63e23-0c15-440c-9e30-3bb576e312f1","Type":"ContainerStarted","Data":"bbc6bc4ac36fda17d35c93979cebf466ec87eb10f44e8f8f6aa9c215aff7ba5c"}
Oct 31 18:40:40.598481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.546074    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-dshm\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.598481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.546230    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-model-cache\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.598481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.546329    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64915137-cd6c-43b7-b45b-c0b958e121a8-tls-certs\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.598481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.546366    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-tmp-dir\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.598481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.546399    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9l8\" (UniqueName: \"kubernetes.io/projected/64915137-cd6c-43b7-b45b-c0b958e121a8-kube-api-access-gg9l8\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.598481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.546524    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-home\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.647995 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.647909    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-model-cache\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.648389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.648005    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64915137-cd6c-43b7-b45b-c0b958e121a8-tls-certs\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.648389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.648038    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-tmp-dir\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.648389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.648073    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9l8\" (UniqueName: \"kubernetes.io/projected/64915137-cd6c-43b7-b45b-c0b958e121a8-kube-api-access-gg9l8\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.648389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.648216    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-home\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.648389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.648284    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-dshm\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.648932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.648432    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-model-cache\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.648932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.648569    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-tmp-dir\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.648932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.648775    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-home\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.677912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.677222    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-dshm\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.678101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.678053    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64915137-cd6c-43b7-b45b-c0b958e121a8-tls-certs\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:40.680235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.680189    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" podStartSLOduration=2.680176103 podStartE2EDuration="2.680176103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:40.678408181 +0000 UTC m=+1143.909646499" watchObservedRunningTime="2026-10-31 18:40:40.680176103 +0000 UTC m=+1143.911414395"
Oct 31 18:40:40.705476 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:40.705436    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9l8\" (UniqueName: \"kubernetes.io/projected/64915137-cd6c-43b7-b45b-c0b958e121a8-kube-api-access-gg9l8\") pod \"distinct-w2-88085e-kserve-dbb778945-9rf8l\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") " pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:41.003633 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:41.003552    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:41.098499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:41.098021    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:41.101856 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:41.101820    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 18:40:41.101973 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:41.101884    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 18:40:41.418983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:41.418947    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"]
Oct 31 18:40:41.420217 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:40:41.420186    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64915137_cd6c_43b7_b45b_c0b958e121a8.slice/crio-a51ee6741c0275d7c107df6ffece1b1ac7cbeb55766023697caaaf5527357bf6 WatchSource:0}: Error finding container a51ee6741c0275d7c107df6ffece1b1ac7cbeb55766023697caaaf5527357bf6: Status 404 returned error can't find the container with id a51ee6741c0275d7c107df6ffece1b1ac7cbeb55766023697caaaf5527357bf6
Oct 31 18:40:41.514800 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:41.514765    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" event={"ID":"64915137-cd6c-43b7-b45b-c0b958e121a8","Type":"ContainerStarted","Data":"a51ee6741c0275d7c107df6ffece1b1ac7cbeb55766023697caaaf5527357bf6"}
Oct 31 18:40:42.097206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:42.097172    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 18:40:42.097410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:42.097240    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 18:40:42.524024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:42.523984    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" event={"ID":"64915137-cd6c-43b7-b45b-c0b958e121a8","Type":"ContainerStarted","Data":"fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960"}
Oct 31 18:40:42.639513 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:42.639450    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podStartSLOduration=2.63943486 podStartE2EDuration="2.63943486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:42.636919185 +0000 UTC m=+1145.868157476" watchObservedRunningTime="2026-10-31 18:40:42.63943486 +0000 UTC m=+1145.870673150"
Oct 31 18:40:43.003827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:43.003779    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:43.096846 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:43.096810    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 18:40:43.097013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:43.096887    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 18:40:44.096828 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:44.096783    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 18:40:44.097320 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:44.096850    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 18:40:45.097314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:45.097275    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 18:40:45.097819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:45.097334    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 18:40:46.096896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:46.096846    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 18:40:46.096896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:46.096898    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 18:40:46.607567 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:40:46.607524    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:40:47.101411 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:47.101384    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:47.101724 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:47.101695    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:47.102362 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:47.102346    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:40:51.005348 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:51.005315    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:51.026656 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:51.026627    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:51.648839 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:51.648805    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:40:54.122487 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:54.121596    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" podUID="35c185c9-da97-4e98-a6e3-36664eba8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:40:54.602596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:54.602546    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:40:54.616927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:40:54.616880    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:13.055705 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.055608    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"]
Oct 31 18:41:13.072484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.072433    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.174489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.162222    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"]
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.334532    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.334597    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfj4\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-kube-api-access-srfj4\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.334660    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.334690    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.334728    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80db6ed8-783b-4782-af33-9e81981246f4-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.334883    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.334952    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80db6ed8-783b-4782-af33-9e81981246f4-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.334980    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.339544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.335019    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.445587    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.445671    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80db6ed8-783b-4782-af33-9e81981246f4-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.445703    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.445737    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.445870    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.445904    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfj4\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-kube-api-access-srfj4\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.445959    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.445987    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.446486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.446022    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80db6ed8-783b-4782-af33-9e81981246f4-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.498510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.450072    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.498510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.450836    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80db6ed8-783b-4782-af33-9e81981246f4-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.498510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.451093    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.498510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.451658    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.498510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.452199    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.590591 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.579117    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80db6ed8-783b-4782-af33-9e81981246f4-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.674160 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.674112    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfj4\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-kube-api-access-srfj4\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.730496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.728205    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:13.746504 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:13.745038    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:14.029480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:14.029424    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:14.473105 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:14.473068    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"]
Oct 31 18:41:14.475337 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:41:14.475304    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80db6ed8_783b_4782_af33_9e81981246f4.slice/crio-49f43258e7960edbe6486d592f1371002f6e9f44a7009fb25e4cd9060718e007 WatchSource:0}: Error finding container 49f43258e7960edbe6486d592f1371002f6e9f44a7009fb25e4cd9060718e007: Status 404 returned error can't find the container with id 49f43258e7960edbe6486d592f1371002f6e9f44a7009fb25e4cd9060718e007
Oct 31 18:41:14.477912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:14.477879    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:14.477998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:14.477944    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:14.477998 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:14.477972    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:14.947542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:14.947507    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" event={"ID":"80db6ed8-783b-4782-af33-9e81981246f4","Type":"ContainerStarted","Data":"7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e"}
Oct 31 18:41:14.947741 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:14.947561    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" event={"ID":"80db6ed8-783b-4782-af33-9e81981246f4","Type":"ContainerStarted","Data":"49f43258e7960edbe6486d592f1371002f6e9f44a7009fb25e4cd9060718e007"}
Oct 31 18:41:15.030382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:15.030349    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:15.031763 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:15.031736    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:15.031887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:15.031784    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 18:41:15.138837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:15.138789    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podStartSLOduration=3.138775568 podStartE2EDuration="3.138775568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:15.136759187 +0000 UTC m=+1178.367997476" watchObservedRunningTime="2026-10-31 18:41:15.138775568 +0000 UTC m=+1178.370013905"
Oct 31 18:41:16.029964 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:16.029926    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:16.030516 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:16.029973    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 18:41:16.614476 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:41:16.614431    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:41:17.030294 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:17.030259    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:17.030812 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:17.030319    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 18:41:18.030563 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:18.030525    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:18.031049 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:18.030580    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 18:41:19.030530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:19.030499    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:19.030736 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:19.030574    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 18:41:20.030864 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:20.030824    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:20.031472 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:20.031418    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 18:41:21.030276 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.030229    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:21.030540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.030296    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 18:41:21.127023 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.126987    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"]
Oct 31 18:41:21.131191 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.131160    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.177185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.177152    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"]
Oct 31 18:41:21.182077 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.182056    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.222666 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.222626    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.222925 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.222907    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.223078 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.223063    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.223368 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.223342    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.223518 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.223415    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.223518 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.223444    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.223617 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.223555    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.223617 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.223585    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-656b4\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-kube-api-access-656b4\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.223708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.223612    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.225619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.225596    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"]
Oct 31 18:41:21.276828 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.276800    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"]
Oct 31 18:41:21.325330 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.325234    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.325535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.325345    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b38d11-b6ac-4102-8b62-d06f4ef93366-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.325535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.325426    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxr7g\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-kube-api-access-lxr7g\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.325672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.325548    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.325672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.325578    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.325672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.325609    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.325801 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.325708    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.326379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326241    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.326379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326017    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.326379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326058    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.326379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326028    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.326379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326323    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.326379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326373    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.326840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326420    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.326840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326657    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.326840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326691    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.326840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326758    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.326840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326813    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.327110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326869    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-656b4\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-kube-api-access-656b4\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.327110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326904    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.327110 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.326984    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.327572 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.327544    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.327691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.327592    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.328825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.328794    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.329497 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.329476    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.383591 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.383557    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.384060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.384032    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-656b4\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-kube-api-access-656b4\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.428414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428382    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.428580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428427    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b38d11-b6ac-4102-8b62-d06f4ef93366-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.428580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428503    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxr7g\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-kube-api-access-lxr7g\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.428580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428541    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.428711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428590    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.428711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428625    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.428711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428649    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.428711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428684    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.428910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428787    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.429141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.428957    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.429141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.429102    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.429290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.429191    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.429290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.429222    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.429371 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.429332    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b38d11-b6ac-4102-8b62-d06f4ef93366-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.431312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.431285    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.431425 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.431405    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.485323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.485289    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.485510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.485327    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxr7g\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-kube-api-access-lxr7g\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.497133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.497111    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:21.747146 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.747100    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:21.949245 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.949213    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"]
Oct 31 18:41:21.949568 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:41:21.949540    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f1579e_c88a_45f9_8130_ddb44b5ace3e.slice/crio-573bbf09e438b44bc385f14a3413447abfdf97c326932313d676441df073234d WatchSource:0}: Error finding container 573bbf09e438b44bc385f14a3413447abfdf97c326932313d676441df073234d: Status 404 returned error can't find the container with id 573bbf09e438b44bc385f14a3413447abfdf97c326932313d676441df073234d
Oct 31 18:41:21.952938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.952898    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:21.953009 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.952990    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:21.953054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:21.953037    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:22.032899 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.032812    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" event={"ID":"76f1579e-c88a-45f9-8130-ddb44b5ace3e","Type":"ContainerStarted","Data":"4d398d7e8fd431aa2e136f169d0ce37d85c1b409c43b069d0c8cc75573e87431"}
Oct 31 18:41:22.032899 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.032871    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" event={"ID":"76f1579e-c88a-45f9-8130-ddb44b5ace3e","Type":"ContainerStarted","Data":"573bbf09e438b44bc385f14a3413447abfdf97c326932313d676441df073234d"}
Oct 31 18:41:22.142914 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.142876    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"]
Oct 31 18:41:22.153803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.153750    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podStartSLOduration=1.153689797 podStartE2EDuration="1.153689797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:22.152360675 +0000 UTC m=+1185.383598965" watchObservedRunningTime="2026-10-31 18:41:22.153689797 +0000 UTC m=+1185.384928088"
Oct 31 18:41:22.166828 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.166788    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:22.166977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.166878    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:22.166977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.166925    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:22.187377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.184564    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:22.187377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.185111    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:22.187377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.186510    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:41:22.497903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:22.497862    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:23.045187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:23.045156    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" event={"ID":"59b38d11-b6ac-4102-8b62-d06f4ef93366","Type":"ContainerStarted","Data":"52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6"}
Oct 31 18:41:23.045338 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:23.045206    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" event={"ID":"59b38d11-b6ac-4102-8b62-d06f4ef93366","Type":"ContainerStarted","Data":"8b40552c0cdb4af04adde6a1db3b48327e9f7239fc06a92a5c2a1df5a7a21ca7"}
Oct 31 18:41:23.189834 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:23.189758    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podStartSLOduration=2.189737583 podStartE2EDuration="2.189737583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:23.167452018 +0000 UTC m=+1186.398690309" watchObservedRunningTime="2026-10-31 18:41:23.189737583 +0000 UTC m=+1186.420975873"
Oct 31 18:41:23.509132 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:23.508753    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 18:41:23.509132 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:23.508817    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 18:41:23.747478 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:23.747422    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:23.749118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:23.749090    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:23.749280 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:23.749150    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 18:41:24.501937 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:24.501895    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 18:41:24.502581 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:24.502546    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 18:41:24.747987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:24.747957    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:24.748175 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:24.748004    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 18:41:25.501050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:25.501011    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 18:41:25.501344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:25.501316    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 18:41:25.748309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:25.748258    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:25.748873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:25.748337    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 18:41:26.498486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:26.497846    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 18:41:26.498486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:26.497912    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 18:41:26.748226 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:26.748194    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:26.748414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:26.748259    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 18:41:27.498485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:27.497872    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 18:41:27.498485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:27.497933    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 18:41:27.747754 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:27.747714    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:27.747929 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:27.747776    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 18:41:28.498654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:28.498612    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 18:41:28.499083 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:28.498684    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 18:41:28.748112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:28.748072    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:28.748255 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:28.748143    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 18:41:29.635567 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:29.635532    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:29.636238 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:29.636209    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:29.637364 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:29.637334    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:41:29.880721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:29.880683    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:29.881037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:29.881011    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:29.881863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:29.881845    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:41:34.139483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:34.138563    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" podUID="35c185c9-da97-4e98-a6e3-36664eba8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:34.621540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:34.620642    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:34.629478 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:34.621892    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.142486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:35.140545    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" podUID="35c185c9-da97-4e98-a6e3-36664eba8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.142486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:35.141342    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:41:35.633483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:35.629543    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.659484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:35.656582    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Oct 31 18:41:35.659484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:35.657510    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:41:35.659484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:35.657574    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:41:35.937540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:35.936077    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:41:36.235540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:36.231124    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:41:36.235540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:36.231217    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:36.669974 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:36.669927    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:36.687766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:36.670613    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:36.945579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:36.944873    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" podUID="35c185c9-da97-4e98-a6e3-36664eba8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 18:41:37.714489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:37.695566    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:37.742632 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:37.726729    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Oct 31 18:41:38.079541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:38.079195    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" podUID="35c185c9-da97-4e98-a6e3-36664eba8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:38.084476 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:38.080614    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:41:38.628488 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:38.623818    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:41:38.663131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:38.663093    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:41:38.663359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:38.663333    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:38.703691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:38.703644    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:39.236483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:39.225817    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86"
Oct 31 18:41:39.715485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:39.694035    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:41:39.742890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:39.742848    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 31 18:41:39.800492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:39.799953    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:41:40.185938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:40.185590    5551 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:40.451880 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:40.451792    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:41:44.522291 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.519002    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"]
Oct 31 18:41:44.548499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.546222    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.617551 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.609851    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3417b822-openshift-default-dockercfg-hx6x8\""
Oct 31 18:41:44.705947 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.705915    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-242k5\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-kube-api-access-242k5\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.706237 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.706204    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.706421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.706393    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.706633 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.706619    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.706889 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.706871    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.712502 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.708569    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.712502 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.708688    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.712502 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.708714    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.712502 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.708824    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d1768a-b31b-40e3-b555-6e12d0542b5a-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.716550 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.716528    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"]
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.834590    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d1768a-b31b-40e3-b555-6e12d0542b5a-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.834691    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-242k5\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-kube-api-access-242k5\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.834746    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.834819    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.834908    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.834936    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.834983    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.835085    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.835107    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.835950    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.836761    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d1768a-b31b-40e3-b555-6e12d0542b5a-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.841954    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.855529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.842392    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.940766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.890232    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.940766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.929143    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-242k5\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-kube-api-access-242k5\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.940766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.932659    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.940766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.933597    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:44.973558 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:44.971838    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:45.226520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:45.226486    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:45.684689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:45.684656    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"]
Oct 31 18:41:45.686675 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:41:45.686651    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d1768a_b31b_40e3_b555_6e12d0542b5a.slice/crio-59b6c38d96ff431165d07926fb460828f6a7960ac8da0bbddb6f802b1750eaa3 WatchSource:0}: Error finding container 59b6c38d96ff431165d07926fb460828f6a7960ac8da0bbddb6f802b1750eaa3: Status 404 returned error can't find the container with id 59b6c38d96ff431165d07926fb460828f6a7960ac8da0bbddb6f802b1750eaa3
Oct 31 18:41:45.689847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:45.689803    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:45.689924 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:45.689900    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:45.689961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:45.689942    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:41:46.320567 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:46.320521    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" event={"ID":"46d1768a-b31b-40e3-b555-6e12d0542b5a","Type":"ContainerStarted","Data":"7958d5da7886ce2650e8603a0ebbdabd309dbe74fab04f85b015c3a6856569de"}
Oct 31 18:41:46.320747 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:46.320595    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" event={"ID":"46d1768a-b31b-40e3-b555-6e12d0542b5a","Type":"ContainerStarted","Data":"59b6c38d96ff431165d07926fb460828f6a7960ac8da0bbddb6f802b1750eaa3"}
Oct 31 18:41:46.488376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:46.488323    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" podStartSLOduration=2.488309646 podStartE2EDuration="2.488309646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:46.486119116 +0000 UTC m=+1209.717357406" watchObservedRunningTime="2026-10-31 18:41:46.488309646 +0000 UTC m=+1209.719547935"
Oct 31 18:41:46.613068 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:41:46.612975    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:41:47.226746 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:47.226707    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:47.232713 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:47.232689    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:47.332445 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:47.332406    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:47.341562 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:47.341512    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:41:53.114481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.103814    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"]
Oct 31 18:41:53.114481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.109540    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:53.154167 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.150712    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"]
Oct 31 18:41:53.199479 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.199247    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3417b822-dockercfg-dgl9j\""
Oct 31 18:41:53.214528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.199962    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3417b822\""
Oct 31 18:41:53.214528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.201957    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6b5522-274e-41c6-a6b3-487b329f4aa2-plugins-config-volume\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9\" (UID: \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:53.214528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.202128    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sv2t\" (UniqueName: \"kubernetes.io/projected/ad6b5522-274e-41c6-a6b3-487b329f4aa2-kube-api-access-8sv2t\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9\" (UID: \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:53.303557 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.303505    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6b5522-274e-41c6-a6b3-487b329f4aa2-plugins-config-volume\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9\" (UID: \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:53.303737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.303690    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sv2t\" (UniqueName: \"kubernetes.io/projected/ad6b5522-274e-41c6-a6b3-487b329f4aa2-kube-api-access-8sv2t\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9\" (UID: \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:53.304278 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.304259    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6b5522-274e-41c6-a6b3-487b329f4aa2-plugins-config-volume\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9\" (UID: \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:53.361231 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.361194    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sv2t\" (UniqueName: \"kubernetes.io/projected/ad6b5522-274e-41c6-a6b3-487b329f4aa2-kube-api-access-8sv2t\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9\" (UID: \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:53.523361 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.523329    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:53.905785 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:53.905748    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"]
Oct 31 18:41:53.906858 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:41:53.906828    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6b5522_274e_41c6_a6b3_487b329f4aa2.slice/crio-11bdaf5a89ca861a96890c30f6445d62759815d20e15a6fea95542165e5d90f3 WatchSource:0}: Error finding container 11bdaf5a89ca861a96890c30f6445d62759815d20e15a6fea95542165e5d90f3: Status 404 returned error can't find the container with id 11bdaf5a89ca861a96890c30f6445d62759815d20e15a6fea95542165e5d90f3
Oct 31 18:41:54.420609 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:54.420564    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9" event={"ID":"ad6b5522-274e-41c6-a6b3-487b329f4aa2","Type":"ContainerStarted","Data":"0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514"}
Oct 31 18:41:54.421173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:54.420677    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9" event={"ID":"ad6b5522-274e-41c6-a6b3-487b329f4aa2","Type":"ContainerStarted","Data":"11bdaf5a89ca861a96890c30f6445d62759815d20e15a6fea95542165e5d90f3"}
Oct 31 18:41:54.421173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:54.420711    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:41:54.534356 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:54.534287    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9" podStartSLOduration=1.534268065 podStartE2EDuration="1.534268065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:54.533508466 +0000 UTC m=+1217.764746756" watchObservedRunningTime="2026-10-31 18:41:54.534268065 +0000 UTC m=+1217.765506356"
Oct 31 18:41:58.413489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.411730    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"]
Oct 31 18:41:58.420483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.417881    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.670902    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.671004    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.671053    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.671099    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.671161    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.671200    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.671237    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2l77\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-kube-api-access-t2l77\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.671340    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.671535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.671380    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.783486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.763525    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"]
Oct 31 18:41:58.825484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.797873    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.831663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.825786    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.855627 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.855592    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2l77\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-kube-api-access-t2l77\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.856007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.855984    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.856478 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.856435    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:58.856758 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:58.856726    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.041548 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.013391    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.206520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.185629    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.206520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.185718    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.206520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.071347    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.206520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.115306    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.206520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.125766    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.206520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.045614    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.206520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.186296    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.206520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.045755    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.207915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.207886    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2l77\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-kube-api-access-t2l77\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.208204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.208164    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.257481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.244124    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.361898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.361792    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:41:59.717711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:41:59.717660    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:00.110642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.110577    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Oct 31 18:42:00.202533 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.184994    5551 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:00.231710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.229950    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:42:00.231710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.230017    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:00.644384 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.644209    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"]
Oct 31 18:42:00.716603 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.716554    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:42:00.716848 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.716800    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:00.961484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.936944    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:42:00.961484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.937030    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:00.961484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.957634    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:42:00.961484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.957737    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:42:00.961484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.957784    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:42:00.962613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.962584    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:00.978530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.976856    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:42:00.978530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:00.976918    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:01.061339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.061307    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:42:01.115499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.113527    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Oct 31 18:42:01.115499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.113910    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:42:01.138663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.138544    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:42:01.138663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.138600    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:01.256275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.256186    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:42:01.256528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.256499    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:01.477008 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.471071    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:42:01.509554 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.508700    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:42:01.716499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:01.711734    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" event={"ID":"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee","Type":"ContainerStarted","Data":"732a9e9de13dcb6c29a38ce2ffc5da80dfb4db1c7770f8e75da7249c32f986e9"}
Oct 31 18:42:02.784484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:02.764813    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" event={"ID":"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee","Type":"ContainerStarted","Data":"515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9"}
Oct 31 18:42:03.043559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:03.043291    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podStartSLOduration=5.043271419 podStartE2EDuration="5.043271419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:03.003133217 +0000 UTC m=+1226.234371545" watchObservedRunningTime="2026-10-31 18:42:03.043271419 +0000 UTC m=+1226.274509714"
Oct 31 18:42:03.371146 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:03.371044    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:42:03.371333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:03.371198    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 18:42:03.371333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:03.371245    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 18:42:04.365488 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:04.363973    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 18:42:04.365488 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:04.364057    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 18:42:05.364898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:05.364860    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 18:42:05.365210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:05.365179    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 18:42:05.431142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:05.431115    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:42:06.371913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:06.371873    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 18:42:06.372221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:06.372187    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 18:42:07.364846 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:07.364807    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 18:42:07.365478 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:07.365423    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 18:42:08.370918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:08.370871    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 18:42:08.371562 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:08.371527    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 18:42:09.363757 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:09.363723    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 18:42:09.363926 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:09.363776    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 18:42:10.367948 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:10.367910    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:42:10.368364 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:10.368312    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:42:10.368999 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:10.368983    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:42:16.604896 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:42:16.604860    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:42:16.898824 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:16.898788    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"]
Oct 31 18:42:16.910306 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:16.910277    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:16.958346 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:16.958317    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"]
Oct 31 18:42:17.002492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.002449    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3417b822-serving-cert\""
Oct 31 18:42:17.031646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.031612    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kj6\" (UniqueName: \"kubernetes.io/projected/caf76a82-fac2-44c6-bb38-514958e44c06-kube-api-access-h2kj6\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.031919 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.031895    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf76a82-fac2-44c6-bb38-514958e44c06-openshift-service-ca\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.032138 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.032117    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-metrics-certs\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.032606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.032581    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.133793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.133758    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf76a82-fac2-44c6-bb38-514958e44c06-openshift-service-ca\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.133977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.133830    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-metrics-certs\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.133977 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.133945    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.134107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.134028    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kj6\" (UniqueName: \"kubernetes.io/projected/caf76a82-fac2-44c6-bb38-514958e44c06-kube-api-access-h2kj6\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.134107 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:42:17.134076    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-serving-cert: secret "maas-api-e2e-mt-3417b822-serving-cert" not found
Oct 31 18:42:17.134215 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:42:17.134148    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls podName:caf76a82-fac2-44c6-bb38-514958e44c06 nodeName:}" failed. No retries permitted until 2026-10-31 18:42:17.634131253 +0000 UTC m=+1240.865369522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls") pod "maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2" (UID: "caf76a82-fac2-44c6-bb38-514958e44c06") : secret "maas-api-e2e-mt-3417b822-serving-cert" not found
Oct 31 18:42:17.135434 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.135399    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf76a82-fac2-44c6-bb38-514958e44c06-openshift-service-ca\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.137585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.137558    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-metrics-certs\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.193700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.193624    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kj6\" (UniqueName: \"kubernetes.io/projected/caf76a82-fac2-44c6-bb38-514958e44c06-kube-api-access-h2kj6\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.640692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.640658    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.643207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.643186    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:17.824797 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:17.824759    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:18.211611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:18.211586    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"]
Oct 31 18:42:18.214303 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:42:18.214280    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf76a82_fac2_44c6_bb38_514958e44c06.slice/crio-9cfe179002ce6cc7f297fb183b288be057fd14456e02d1792baacc5d14649afc WatchSource:0}: Error finding container 9cfe179002ce6cc7f297fb183b288be057fd14456e02d1792baacc5d14649afc: Status 404 returned error can't find the container with id 9cfe179002ce6cc7f297fb183b288be057fd14456e02d1792baacc5d14649afc
Oct 31 18:42:18.215173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:18.215155    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:42:18.916177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:18.916125    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2" event={"ID":"caf76a82-fac2-44c6-bb38-514958e44c06","Type":"ContainerStarted","Data":"9cfe179002ce6cc7f297fb183b288be057fd14456e02d1792baacc5d14649afc"}
Oct 31 18:42:19.928076 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:19.928034    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2" event={"ID":"caf76a82-fac2-44c6-bb38-514958e44c06","Type":"ContainerStarted","Data":"091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea"}
Oct 31 18:42:19.928570 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:19.928176    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:20.044343 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:20.044294    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2" podStartSLOduration=3.184559995 podStartE2EDuration="4.044279131s" totalImagesPullingTime="859.719136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:20.041859826 +0000 UTC m=+1243.273098109" watchObservedRunningTime="2026-10-31 18:42:20.044279131 +0000 UTC m=+1243.275517421"
Oct 31 18:42:25.940495 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:25.940445    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:42:34.971750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:34.970564    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" podUID="35c185c9-da97-4e98-a6e3-36664eba8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:35.527540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:35.521625    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:35.527540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:35.523544    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:35.614978 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:35.614934    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"]
Oct 31 18:42:36.208627 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:36.208584    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:42:36.209249 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:36.209210    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:36.522638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:36.522539    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:36.523552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:36.523529    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:42:36.525322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:36.523954    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:36.525833 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:36.525816    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:42:37.020515 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:36.988818    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 18:42:37.399538 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:37.379719    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 18:42:40.615657 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:40.615611    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb" podUID="0e9bef38-5973-4332-aa0a-d826734112ba" containerName="payload-processing" containerID="cri-o://c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40" gracePeriod=25
Oct 31 18:42:40.925729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:40.925703    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:42:41.026281 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.026245    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-plugins-config-volume\") pod \"0e9bef38-5973-4332-aa0a-d826734112ba\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") "
Oct 31 18:42:41.026501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.026355    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-routing-overlay-volume\") pod \"0e9bef38-5973-4332-aa0a-d826734112ba\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") "
Oct 31 18:42:41.026501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.026392    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-provider-credentials\") pod \"0e9bef38-5973-4332-aa0a-d826734112ba\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") "
Oct 31 18:42:41.026635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.026504    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-kube-api-access-n2sx4\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-kube-api-access-n2sx4\") pod \"0e9bef38-5973-4332-aa0a-d826734112ba\" (UID: \"0e9bef38-5973-4332-aa0a-d826734112ba\") "
Oct 31 18:42:41.026741 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.026712    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-plugins-config-volume" pod "0e9bef38-5973-4332-aa0a-d826734112ba" (UID: "0e9bef38-5973-4332-aa0a-d826734112ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:42:41.026827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.026802    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-routing-overlay-volume" pod "0e9bef38-5973-4332-aa0a-d826734112ba" (UID: "0e9bef38-5973-4332-aa0a-d826734112ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:42:41.027101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.027084    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:42:41.027203 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.027103    5551 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e9bef38-5973-4332-aa0a-d826734112ba-routing-overlay-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:42:41.029776 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.029751    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-kube-api-access-n2sx4" pod "0e9bef38-5973-4332-aa0a-d826734112ba" (UID: "0e9bef38-5973-4332-aa0a-d826734112ba"). InnerVolumeSpecName "kube-api-access-n2sx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:42:41.030005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.029983    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-provider-credentials" pod "0e9bef38-5973-4332-aa0a-d826734112ba" (UID: "0e9bef38-5973-4332-aa0a-d826734112ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:42:41.128115 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.128081    5551 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-provider-credentials\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:42:41.128115 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.128108    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2sx4\" (UniqueName: \"kubernetes.io/projected/0e9bef38-5973-4332-aa0a-d826734112ba-kube-api-access-n2sx4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:42:41.582445 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.582401    5551 generic.go:356] "Generic (PLEG): container finished" podID="0e9bef38-5973-4332-aa0a-d826734112ba" containerID="c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40" exitCode=0
Oct 31 18:42:41.582668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.582496    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"
Oct 31 18:42:41.582668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.582493    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb" event={"ID":"0e9bef38-5973-4332-aa0a-d826734112ba","Type":"ContainerDied","Data":"c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40"}
Oct 31 18:42:41.582668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.582557    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb" event={"ID":"0e9bef38-5973-4332-aa0a-d826734112ba","Type":"ContainerDied","Data":"f63999e15509480995896f625c9dbff64038aa4dc36eee3892d8d8231e612a09"}
Oct 31 18:42:41.582668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.582581    5551 scope.go:122] "RemoveContainer" containerID="c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40"
Oct 31 18:42:41.594575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.594553    5551 scope.go:122] "RemoveContainer" containerID="c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40"
Oct 31 18:42:41.594853 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:42:41.594835    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40\": container with ID starting with c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40 not found: ID does not exist" containerID="c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40"
Oct 31 18:42:41.594897 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.594863    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40"} err="failed to get container status \"c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40\": rpc error: code = NotFound desc = could not find container \"c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40\": container with ID starting with c37fecd45e45f1e4d62aba2e5a6b59381b17a68666fdd395436bd27253da9f40 not found: ID does not exist"
Oct 31 18:42:41.748863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.748829    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"]
Oct 31 18:42:41.796359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:41.796330    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7869f7dc8d-plpcb"]
Oct 31 18:42:42.357342 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.357303    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"]
Oct 31 18:42:42.358276 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.358253    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0e9bef38-5973-4332-aa0a-d826734112ba" containerName="payload-processing"
Oct 31 18:42:42.358334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.358282    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9bef38-5973-4332-aa0a-d826734112ba" containerName="payload-processing"
Oct 31 18:42:42.358529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.358515    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9bef38-5973-4332-aa0a-d826734112ba" containerName="payload-processing"
Oct 31 18:42:42.364034 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.364010    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:42.414910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.414874    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"]
Oct 31 18:42:42.415981 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.415949    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bd6c0a\""
Oct 31 18:42:42.416108 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.416062    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bd6c0a-dockercfg-ksxnd\""
Oct 31 18:42:42.463489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.455405    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4acb6dcf-ec1e-4972-84f2-8372cd240632-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67\" (UID: \"4acb6dcf-ec1e-4972-84f2-8372cd240632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:42.465478 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.464091    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snt6s\" (UniqueName: \"kubernetes.io/projected/4acb6dcf-ec1e-4972-84f2-8372cd240632-kube-api-access-snt6s\") pod \"payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67\" (UID: \"4acb6dcf-ec1e-4972-84f2-8372cd240632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:42.593528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.591824    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snt6s\" (UniqueName: \"kubernetes.io/projected/4acb6dcf-ec1e-4972-84f2-8372cd240632-kube-api-access-snt6s\") pod \"payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67\" (UID: \"4acb6dcf-ec1e-4972-84f2-8372cd240632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:42.593528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.591976    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4acb6dcf-ec1e-4972-84f2-8372cd240632-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67\" (UID: \"4acb6dcf-ec1e-4972-84f2-8372cd240632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:42.593528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.592872    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4acb6dcf-ec1e-4972-84f2-8372cd240632-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67\" (UID: \"4acb6dcf-ec1e-4972-84f2-8372cd240632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:42.626624 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.621884    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"]
Oct 31 18:42:42.626624 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.626383    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:42.626624 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.626409    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"]
Oct 31 18:42:42.649894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.649862    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snt6s\" (UniqueName: \"kubernetes.io/projected/4acb6dcf-ec1e-4972-84f2-8372cd240632-kube-api-access-snt6s\") pod \"payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67\" (UID: \"4acb6dcf-ec1e-4972-84f2-8372cd240632\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:42.678764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.678738    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:42.693655 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.693631    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056e6a86-6619-4821-96e2-d43e08a1dd37-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk\" (UID: \"056e6a86-6619-4821-96e2-d43e08a1dd37\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:42.693788 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.693770    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxztz\" (UniqueName: \"kubernetes.io/projected/056e6a86-6619-4821-96e2-d43e08a1dd37-kube-api-access-jxztz\") pod \"payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk\" (UID: \"056e6a86-6619-4821-96e2-d43e08a1dd37\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:42.795254 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.795216    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxztz\" (UniqueName: \"kubernetes.io/projected/056e6a86-6619-4821-96e2-d43e08a1dd37-kube-api-access-jxztz\") pod \"payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk\" (UID: \"056e6a86-6619-4821-96e2-d43e08a1dd37\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:42.795723 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.795477    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056e6a86-6619-4821-96e2-d43e08a1dd37-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk\" (UID: \"056e6a86-6619-4821-96e2-d43e08a1dd37\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:42.796206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.796182    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056e6a86-6619-4821-96e2-d43e08a1dd37-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk\" (UID: \"056e6a86-6619-4821-96e2-d43e08a1dd37\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:42.852516 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.852482    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxztz\" (UniqueName: \"kubernetes.io/projected/056e6a86-6619-4821-96e2-d43e08a1dd37-kube-api-access-jxztz\") pod \"payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk\" (UID: \"056e6a86-6619-4821-96e2-d43e08a1dd37\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:42.910364 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.910291    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9bef38-5973-4332-aa0a-d826734112ba" path="/var/lib/kubelet/pods/0e9bef38-5973-4332-aa0a-d826734112ba/volumes"
Oct 31 18:42:42.945207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:42.945180    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:43.097054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.097023    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"]
Oct 31 18:42:43.098122 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:42:43.098026    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acb6dcf_ec1e_4972_84f2_8372cd240632.slice/crio-e47ffbbc29e207fb4bdaf4a22bd6760c6c766383b1cabac7da26b6301114bb49 WatchSource:0}: Error finding container e47ffbbc29e207fb4bdaf4a22bd6760c6c766383b1cabac7da26b6301114bb49: Status 404 returned error can't find the container with id e47ffbbc29e207fb4bdaf4a22bd6760c6c766383b1cabac7da26b6301114bb49
Oct 31 18:42:43.377012 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.376974    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"]
Oct 31 18:42:43.378832 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:42:43.378799    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056e6a86_6619_4821_96e2_d43e08a1dd37.slice/crio-0c5a48487be37b220e02355c089f4719ace6d8633e71d4d6fb0a08581bf00b75 WatchSource:0}: Error finding container 0c5a48487be37b220e02355c089f4719ace6d8633e71d4d6fb0a08581bf00b75: Status 404 returned error can't find the container with id 0c5a48487be37b220e02355c089f4719ace6d8633e71d4d6fb0a08581bf00b75
Oct 31 18:42:43.643178 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.643140    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk" event={"ID":"056e6a86-6619-4821-96e2-d43e08a1dd37","Type":"ContainerStarted","Data":"6d6d32a2e2ed406f11777ef7322f49d9df7aaadef48cea305243f0c2c3a6396c"}
Oct 31 18:42:43.643530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.643498    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk" event={"ID":"056e6a86-6619-4821-96e2-d43e08a1dd37","Type":"ContainerStarted","Data":"0c5a48487be37b220e02355c089f4719ace6d8633e71d4d6fb0a08581bf00b75"}
Oct 31 18:42:43.643681 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.643547    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:43.645255 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.645222    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67" event={"ID":"4acb6dcf-ec1e-4972-84f2-8372cd240632","Type":"ContainerStarted","Data":"123867a64a15c1c82cc4ae84dda69ac030b80dbef74718e61d4ce2bea963cd10"}
Oct 31 18:42:43.645379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.645286    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:43.645379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.645310    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67" event={"ID":"4acb6dcf-ec1e-4972-84f2-8372cd240632","Type":"ContainerStarted","Data":"e47ffbbc29e207fb4bdaf4a22bd6760c6c766383b1cabac7da26b6301114bb49"}
Oct 31 18:42:43.760984 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.760205    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk" podStartSLOduration=1.7601691449999999 podStartE2EDuration="1.760169145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:43.755616405 +0000 UTC m=+1266.986854694" watchObservedRunningTime="2026-10-31 18:42:43.760169145 +0000 UTC m=+1266.991407436"
Oct 31 18:42:43.879646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:43.879575    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67" podStartSLOduration=1.879553106 podStartE2EDuration="1.879553106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:43.873888016 +0000 UTC m=+1267.105126307" watchObservedRunningTime="2026-10-31 18:42:43.879553106 +0000 UTC m=+1267.110791396"
Oct 31 18:42:45.546928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.546892    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"]
Oct 31 18:42:45.551440 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.551415    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.603748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.603716    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 18:42:45.603748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.603719    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 18:42:45.604024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.603803    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 18:42:45.637592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.637563    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.637717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.637602    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.637717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.637628    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1178707-69aa-42df-9675-74cf68d4eef5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.637851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.637779    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.637851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.637823    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.637936 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.637913    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.637984 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.637957    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdhg\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-kube-api-access-9zdhg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.656148 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.654000    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vkrfg\""
Oct 31 18:42:45.661020 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.660147    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"]
Oct 31 18:42:45.739037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.739004    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.739187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.739055    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.739187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.739113    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.739187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.739136    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdhg\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-kube-api-access-9zdhg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.739340 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.739255    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.739340 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.739309    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.739641 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.739590    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1178707-69aa-42df-9675-74cf68d4eef5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.740215 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.740130    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.741797 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.741763    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1178707-69aa-42df-9675-74cf68d4eef5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.741903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.741808    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.741903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.741844    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.742014 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.741902    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.796089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.796060    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.796234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.796215    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdhg\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-kube-api-access-9zdhg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-897nc\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:45.867182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:45.867118    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:46.301179 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:42:46.301147    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1178707_69aa_42df_9675_74cf68d4eef5.slice/crio-e97b74637cbd903793c907d9183dcb63cfc84719c0886a9b8f2f3c6e810971b8 WatchSource:0}: Error finding container e97b74637cbd903793c907d9183dcb63cfc84719c0886a9b8f2f3c6e810971b8: Status 404 returned error can't find the container with id e97b74637cbd903793c907d9183dcb63cfc84719c0886a9b8f2f3c6e810971b8
Oct 31 18:42:46.301922 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:46.301886    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"]
Oct 31 18:42:46.603963 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:42:46.603879    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:42:46.679093 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:46.679056    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc" event={"ID":"d1178707-69aa-42df-9675-74cf68d4eef5","Type":"ContainerStarted","Data":"e97b74637cbd903793c907d9183dcb63cfc84719c0886a9b8f2f3c6e810971b8"}
Oct 31 18:42:50.012693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:50.012664    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:42:50.012989 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:50.012724    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:42:50.720081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:50.720035    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc" event={"ID":"d1178707-69aa-42df-9675-74cf68d4eef5","Type":"ContainerStarted","Data":"35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e"}
Oct 31 18:42:50.720566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:50.720542    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:42:50.938894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:50.938844    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc" podStartSLOduration=2.230735444 podStartE2EDuration="5.938832683s" totalImagesPullingTime="3.708097239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:50.889174304 +0000 UTC m=+1274.120412605" watchObservedRunningTime="2026-10-31 18:42:50.938832683 +0000 UTC m=+1274.170070972"
Oct 31 18:42:51.730329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:51.730289    5551 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-897nc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:42:51.730747 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:51.730359    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc" podUID="d1178707-69aa-42df-9675-74cf68d4eef5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:42:54.658038 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:54.658006    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:42:54.658582 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:54.658284    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:42:54.729268 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:42:54.729240    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:43:01.102486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:01.100610    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86" podUID="35c185c9-da97-4e98-a6e3-36664eba8faf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:43:01.238093 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:01.237946    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:43:01.275677 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:01.275540    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:43:01.275677 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:01.275602    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:43:07.392688 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:07.392651    5551 generic.go:356] "Generic (PLEG): container finished" podID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerID="7958d5da7886ce2650e8603a0ebbdabd309dbe74fab04f85b015c3a6856569de" exitCode=0
Oct 31 18:43:07.393096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:07.392714    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" event={"ID":"46d1768a-b31b-40e3-b555-6e12d0542b5a","Type":"ContainerDied","Data":"7958d5da7886ce2650e8603a0ebbdabd309dbe74fab04f85b015c3a6856569de"}
Oct 31 18:43:07.393137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:07.393101    5551 scope.go:122] "RemoveContainer" containerID="7958d5da7886ce2650e8603a0ebbdabd309dbe74fab04f85b015c3a6856569de"
Oct 31 18:43:07.394690 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:07.394659    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:43:07.394766 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:07.394750    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:43:07.394814 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:07.394802    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:43:08.401413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:08.401375    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" event={"ID":"46d1768a-b31b-40e3-b555-6e12d0542b5a","Type":"ContainerStarted","Data":"e7fb9169eb0dd25cb5e2fca2fd2497747e742175dd9694090fabe6cd5fbb05fa"}
Oct 31 18:43:09.227743 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:09.227695    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:43:09.228009 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:09.227984    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:43:09.228120 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:09.228035    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 18:43:10.227162 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:10.227124    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:43:10.227563 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:10.227194    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 18:43:11.226931 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:11.226893    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:43:11.227135 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:11.226947    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 18:43:12.227736 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:12.227706    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:43:12.228136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:12.227756    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 18:43:13.227029 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:13.226991    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:43:13.227214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:13.227044    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 18:43:14.231118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:14.231086    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:43:14.231532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:14.231512    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:43:14.232310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:43:14.232291    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:43:16.610842 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:43:16.610810    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:43:46.606629 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:43:46.606581    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:44:16.609288 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:44:16.609091    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:44:46.604819 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:44:46.604783    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:45:00.166764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.166731    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"]
Oct 31 18:45:00.171710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.171689    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:00.187837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.187814    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"]
Oct 31 18:45:00.192166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.192147    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:00.228910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.228879    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8c690-5fb3-4e72-aa60-840831e184bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh\" (UID: \"e0d8c690-5fb3-4e72-aa60-840831e184bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:00.229042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.229020    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ds9\" (UniqueName: \"kubernetes.io/projected/e0d8c690-5fb3-4e72-aa60-840831e184bc-kube-api-access-r4ds9\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh\" (UID: \"e0d8c690-5fb3-4e72-aa60-840831e184bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:00.265654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.265630    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"]
Oct 31 18:45:00.326560 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.326191    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:45:00.330481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.330252    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8c690-5fb3-4e72-aa60-840831e184bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh\" (UID: \"e0d8c690-5fb3-4e72-aa60-840831e184bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:00.330481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.330364    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ds9\" (UniqueName: \"kubernetes.io/projected/e0d8c690-5fb3-4e72-aa60-840831e184bc-kube-api-access-r4ds9\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh\" (UID: \"e0d8c690-5fb3-4e72-aa60-840831e184bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:00.330777 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.330522    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc\" (UID: \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:00.330777 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.330554    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxsd\" (UniqueName: \"kubernetes.io/projected/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-kube-api-access-2zxsd\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc\" (UID: \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:00.331094 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.331063    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"]
Oct 31 18:45:00.331668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.331642    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8c690-5fb3-4e72-aa60-840831e184bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh\" (UID: \"e0d8c690-5fb3-4e72-aa60-840831e184bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:00.388190 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.388166    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ds9\" (UniqueName: \"kubernetes.io/projected/e0d8c690-5fb3-4e72-aa60-840831e184bc-kube-api-access-r4ds9\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh\" (UID: \"e0d8c690-5fb3-4e72-aa60-840831e184bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:00.432497 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.432411    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc\" (UID: \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:00.432497 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.432489    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxsd\" (UniqueName: \"kubernetes.io/projected/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-kube-api-access-2zxsd\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc\" (UID: \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:00.432993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.432974    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc\" (UID: \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:00.489064 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.489039    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:00.491894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.491858    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxsd\" (UniqueName: \"kubernetes.io/projected/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-kube-api-access-2zxsd\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc\" (UID: \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:00.507740 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.507703    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:00.714706 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.714633    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"]
Oct 31 18:45:00.719980 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.719956    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.772539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.772443    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:45:00.776674 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.776634    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"]
Oct 31 18:45:00.837302 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.837270    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.837511 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.837489    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwrg\" (UniqueName: \"kubernetes.io/projected/0c0e48de-3861-4641-87d8-91d32549def1-kube-api-access-6pwrg\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.837577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.837561    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.887398 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:45:00.887364    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d8c690_5fb3_4e72_aa60_840831e184bc.slice/crio-f8bef1886ea2808ed35bdb2e90b94da25f0004d63769415707efc2a41a223177 WatchSource:0}: Error finding container f8bef1886ea2808ed35bdb2e90b94da25f0004d63769415707efc2a41a223177: Status 404 returned error can't find the container with id f8bef1886ea2808ed35bdb2e90b94da25f0004d63769415707efc2a41a223177
Oct 31 18:45:00.888380 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.888352    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"]
Oct 31 18:45:00.935277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.935251    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"]
Oct 31 18:45:00.937827 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:45:00.937798    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe060b2_fa68_43b5_b9b9_a2a283dc7626.slice/crio-c333c087e140de5bdb98c994cb268f7efffafdd814b0574fd2e33a89a26e996b WatchSource:0}: Error finding container c333c087e140de5bdb98c994cb268f7efffafdd814b0574fd2e33a89a26e996b: Status 404 returned error can't find the container with id c333c087e140de5bdb98c994cb268f7efffafdd814b0574fd2e33a89a26e996b
Oct 31 18:45:00.938387 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.938364    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwrg\" (UniqueName: \"kubernetes.io/projected/0c0e48de-3861-4641-87d8-91d32549def1-kube-api-access-6pwrg\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.938448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.938430    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.938659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.938641    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.939327 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.939189    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.941269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.941250    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:00.995847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:00.995821    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwrg\" (UniqueName: \"kubernetes.io/projected/0c0e48de-3861-4641-87d8-91d32549def1-kube-api-access-6pwrg\") pod \"maas-api-revoke-sub-7fad6d17-nmjml\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:01.035978 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.035952    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:01.420778 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.420748    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"]
Oct 31 18:45:01.421224 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:45:01.421199    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0e48de_3861_4641_87d8_91d32549def1.slice/crio-4a1b79d2ccc1417ef0d87503cb9c3ac77cc1ac3ea4be24dc46c58c134bf91f6f WatchSource:0}: Error finding container 4a1b79d2ccc1417ef0d87503cb9c3ac77cc1ac3ea4be24dc46c58c134bf91f6f: Status 404 returned error can't find the container with id 4a1b79d2ccc1417ef0d87503cb9c3ac77cc1ac3ea4be24dc46c58c134bf91f6f
Oct 31 18:45:01.581767 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.581684    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" event={"ID":"6fe060b2-fa68-43b5-b9b9-a2a283dc7626","Type":"ContainerStarted","Data":"f82ddb1173214a0622549df30bce91a7e59415ce5bd786226bb61bf1aa350630"}
Oct 31 18:45:01.581767 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.581748    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" event={"ID":"6fe060b2-fa68-43b5-b9b9-a2a283dc7626","Type":"ContainerStarted","Data":"c333c087e140de5bdb98c994cb268f7efffafdd814b0574fd2e33a89a26e996b"}
Oct 31 18:45:01.583092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.583067    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml" event={"ID":"0c0e48de-3861-4641-87d8-91d32549def1","Type":"ContainerStarted","Data":"4a1b79d2ccc1417ef0d87503cb9c3ac77cc1ac3ea4be24dc46c58c134bf91f6f"}
Oct 31 18:45:01.584580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.584555    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" event={"ID":"e0d8c690-5fb3-4e72-aa60-840831e184bc","Type":"ContainerStarted","Data":"5da8d6435c2005a0d26d06a660390c6ddafe61f565a12eb82fc83b17705ab2ac"}
Oct 31 18:45:01.584659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.584609    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" event={"ID":"e0d8c690-5fb3-4e72-aa60-840831e184bc","Type":"ContainerStarted","Data":"f8bef1886ea2808ed35bdb2e90b94da25f0004d63769415707efc2a41a223177"}
Oct 31 18:45:01.697776 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.697719    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" podStartSLOduration=1.697705387 podStartE2EDuration="1.697705387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:01.693962686 +0000 UTC m=+1404.925201001" watchObservedRunningTime="2026-10-31 18:45:01.697705387 +0000 UTC m=+1404.928943677"
Oct 31 18:45:01.809142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:01.809083    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" podStartSLOduration=1.809064496 podStartE2EDuration="1.809064496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:01.806542294 +0000 UTC m=+1405.037780583" watchObservedRunningTime="2026-10-31 18:45:01.809064496 +0000 UTC m=+1405.040302787"
Oct 31 18:45:02.593833 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:02.593373    5551 generic.go:356] "Generic (PLEG): container finished" podID="0c0e48de-3861-4641-87d8-91d32549def1" containerID="df95fd8bbd2e2dfd5ecb75cb0ecb423f64c13d300548aa2ad1dce80dc8ad1131" exitCode=0
Oct 31 18:45:02.594536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:02.593793    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml" event={"ID":"0c0e48de-3861-4641-87d8-91d32549def1","Type":"ContainerDied","Data":"df95fd8bbd2e2dfd5ecb75cb0ecb423f64c13d300548aa2ad1dce80dc8ad1131"}
Oct 31 18:45:02.599033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:02.599004    5551 generic.go:356] "Generic (PLEG): container finished" podID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerID="5da8d6435c2005a0d26d06a660390c6ddafe61f565a12eb82fc83b17705ab2ac" exitCode=60
Oct 31 18:45:02.599185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:02.599085    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" event={"ID":"e0d8c690-5fb3-4e72-aa60-840831e184bc","Type":"ContainerDied","Data":"5da8d6435c2005a0d26d06a660390c6ddafe61f565a12eb82fc83b17705ab2ac"}
Oct 31 18:45:02.599660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:02.599639    5551 scope.go:122] "RemoveContainer" containerID="5da8d6435c2005a0d26d06a660390c6ddafe61f565a12eb82fc83b17705ab2ac"
Oct 31 18:45:02.600924 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:02.600901    5551 generic.go:356] "Generic (PLEG): container finished" podID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerID="f82ddb1173214a0622549df30bce91a7e59415ce5bd786226bb61bf1aa350630" exitCode=60
Oct 31 18:45:02.601022 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:02.600991    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" event={"ID":"6fe060b2-fa68-43b5-b9b9-a2a283dc7626","Type":"ContainerDied","Data":"f82ddb1173214a0622549df30bce91a7e59415ce5bd786226bb61bf1aa350630"}
Oct 31 18:45:02.601231 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:02.601213    5551 scope.go:122] "RemoveContainer" containerID="f82ddb1173214a0622549df30bce91a7e59415ce5bd786226bb61bf1aa350630"
Oct 31 18:45:03.610200 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:03.610105    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" event={"ID":"e0d8c690-5fb3-4e72-aa60-840831e184bc","Type":"ContainerStarted","Data":"820a0ddff9cb4fab40bd0bcfc00ef7e095762f0b68ce893b25f2dd71683f8893"}
Oct 31 18:45:03.612165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:03.612133    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" event={"ID":"6fe060b2-fa68-43b5-b9b9-a2a283dc7626","Type":"ContainerStarted","Data":"6c6ec1b900edf2c16a57d653b7d33879d2b70ae2e9b2e31ab89c20343e64aa40"}
Oct 31 18:45:03.846554 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:03.846525    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:04.001898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.001864    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-xks-ca\") pod \"0c0e48de-3861-4641-87d8-91d32549def1\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") "
Oct 31 18:45:04.002089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.001982    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-openshift-ca\") pod \"0c0e48de-3861-4641-87d8-91d32549def1\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") "
Oct 31 18:45:04.002089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.002078    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0e48de-3861-4641-87d8-91d32549def1-kube-api-access-6pwrg\" (UniqueName: \"kubernetes.io/projected/0c0e48de-3861-4641-87d8-91d32549def1-kube-api-access-6pwrg\") pod \"0c0e48de-3861-4641-87d8-91d32549def1\" (UID: \"0c0e48de-3861-4641-87d8-91d32549def1\") "
Oct 31 18:45:04.002613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.002582    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-openshift-ca" pod "0c0e48de-3861-4641-87d8-91d32549def1" (UID: "0c0e48de-3861-4641-87d8-91d32549def1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:04.002910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.002885    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:04.004372 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.004335    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-xks-ca" pod "0c0e48de-3861-4641-87d8-91d32549def1" (UID: "0c0e48de-3861-4641-87d8-91d32549def1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:45:04.005118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.005100    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0e48de-3861-4641-87d8-91d32549def1-kube-api-access-6pwrg" pod "0c0e48de-3861-4641-87d8-91d32549def1" (UID: "0c0e48de-3861-4641-87d8-91d32549def1"). InnerVolumeSpecName "kube-api-access-6pwrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:04.104847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.104800    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pwrg\" (UniqueName: \"kubernetes.io/projected/0c0e48de-3861-4641-87d8-91d32549def1-kube-api-access-6pwrg\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:04.104847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.104835    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0e48de-3861-4641-87d8-91d32549def1-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:04.621701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.621675    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"
Oct 31 18:45:04.622109 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.621675    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml" event={"ID":"0c0e48de-3861-4641-87d8-91d32549def1","Type":"ContainerDied","Data":"4a1b79d2ccc1417ef0d87503cb9c3ac77cc1ac3ea4be24dc46c58c134bf91f6f"}
Oct 31 18:45:04.622109 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.621808    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a1b79d2ccc1417ef0d87503cb9c3ac77cc1ac3ea4be24dc46c58c134bf91f6f"
Oct 31 18:45:04.623824 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.623803    5551 generic.go:356] "Generic (PLEG): container finished" podID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerID="820a0ddff9cb4fab40bd0bcfc00ef7e095762f0b68ce893b25f2dd71683f8893" exitCode=60
Oct 31 18:45:04.623932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.623888    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" event={"ID":"e0d8c690-5fb3-4e72-aa60-840831e184bc","Type":"ContainerDied","Data":"820a0ddff9cb4fab40bd0bcfc00ef7e095762f0b68ce893b25f2dd71683f8893"}
Oct 31 18:45:04.623991 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.623940    5551 scope.go:122] "RemoveContainer" containerID="5da8d6435c2005a0d26d06a660390c6ddafe61f565a12eb82fc83b17705ab2ac"
Oct 31 18:45:04.624165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.624145    5551 scope.go:122] "RemoveContainer" containerID="820a0ddff9cb4fab40bd0bcfc00ef7e095762f0b68ce893b25f2dd71683f8893"
Oct 31 18:45:04.624539 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:04.624514    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh_odh-ai-gateway-infra(e0d8c690-5fb3-4e72-aa60-840831e184bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc"
Oct 31 18:45:04.626087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.626068    5551 generic.go:356] "Generic (PLEG): container finished" podID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerID="6c6ec1b900edf2c16a57d653b7d33879d2b70ae2e9b2e31ab89c20343e64aa40" exitCode=60
Oct 31 18:45:04.626160 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.626095    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" event={"ID":"6fe060b2-fa68-43b5-b9b9-a2a283dc7626","Type":"ContainerDied","Data":"6c6ec1b900edf2c16a57d653b7d33879d2b70ae2e9b2e31ab89c20343e64aa40"}
Oct 31 18:45:04.626432 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.626413    5551 scope.go:122] "RemoveContainer" containerID="6c6ec1b900edf2c16a57d653b7d33879d2b70ae2e9b2e31ab89c20343e64aa40"
Oct 31 18:45:04.626751 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:04.626729    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc_odh-ai-gateway-infra(6fe060b2-fa68-43b5-b9b9-a2a283dc7626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626"
Oct 31 18:45:04.650637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:04.650382    5551 scope.go:122] "RemoveContainer" containerID="f82ddb1173214a0622549df30bce91a7e59415ce5bd786226bb61bf1aa350630"
Oct 31 18:45:05.644231 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:05.644202    5551 scope.go:122] "RemoveContainer" containerID="820a0ddff9cb4fab40bd0bcfc00ef7e095762f0b68ce893b25f2dd71683f8893"
Oct 31 18:45:05.644695 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:05.644520    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh_odh-ai-gateway-infra(e0d8c690-5fb3-4e72-aa60-840831e184bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc"
Oct 31 18:45:05.645754 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:05.645724    5551 scope.go:122] "RemoveContainer" containerID="6c6ec1b900edf2c16a57d653b7d33879d2b70ae2e9b2e31ab89c20343e64aa40"
Oct 31 18:45:05.645966 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:05.645949    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc_odh-ai-gateway-infra(6fe060b2-fa68-43b5-b9b9-a2a283dc7626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626"
Oct 31 18:45:08.046711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.046677    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"]
Oct 31 18:45:08.047325 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.047312    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0c0e48de-3861-4641-87d8-91d32549def1" containerName="revoke-keys"
Oct 31 18:45:08.047367 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.047327    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0e48de-3861-4641-87d8-91d32549def1" containerName="revoke-keys"
Oct 31 18:45:08.047486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.047475    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0e48de-3861-4641-87d8-91d32549def1" containerName="revoke-keys"
Oct 31 18:45:08.052131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.052109    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.103933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.103903    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6279f5-openshift-default-dockercfg-d8cq6\""
Oct 31 18:45:08.159821 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.159787    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"]
Oct 31 18:45:08.160681 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.160642    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.160837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.160725    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k4dw\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-kube-api-access-5k4dw\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.160837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.160777    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.160967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.160834    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.160967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.160911    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.160967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.160934    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.161129 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.161053    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.161129 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.161078    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.161222 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.161157    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002e88e-f508-445a-8b51-4436e5b4b93b-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262286 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262243    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262294    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262369    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002e88e-f508-445a-8b51-4436e5b4b93b-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262472    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262529    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k4dw\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-kube-api-access-5k4dw\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262579    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262618    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262692    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.262825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.262716    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.264276 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.264238    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.264761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.264737    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.265202 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.265182    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.266207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.265934    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002e88e-f508-445a-8b51-4436e5b4b93b-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.266344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.266159    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.284059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.284021    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.284306 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.284282    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.320612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.320532    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.320612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.320555    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k4dw\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-kube-api-access-5k4dw\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.367954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.367924    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:08.807019 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.806994    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"]
Oct 31 18:45:08.807793 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:45:08.807767    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc002e88e_f508_445a_8b51_4436e5b4b93b.slice/crio-acf77b87dc5818a8e4f6729b03ace1a9d2d093e7f43b81b75c09ec0fa9ebe9a8 WatchSource:0}: Error finding container acf77b87dc5818a8e4f6729b03ace1a9d2d093e7f43b81b75c09ec0fa9ebe9a8: Status 404 returned error can't find the container with id acf77b87dc5818a8e4f6729b03ace1a9d2d093e7f43b81b75c09ec0fa9ebe9a8
Oct 31 18:45:08.810156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.810121    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:45:08.810238 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.810197    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:45:08.810238 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:08.810225    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:45:09.693803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:09.693768    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" event={"ID":"c002e88e-f508-445a-8b51-4436e5b4b93b","Type":"ContainerStarted","Data":"50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41"}
Oct 31 18:45:09.694219 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:09.693823    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" event={"ID":"c002e88e-f508-445a-8b51-4436e5b4b93b","Type":"ContainerStarted","Data":"acf77b87dc5818a8e4f6729b03ace1a9d2d093e7f43b81b75c09ec0fa9ebe9a8"}
Oct 31 18:45:09.897482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:09.880447    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podStartSLOduration=1.880423912 podStartE2EDuration="1.880423912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:09.87828712 +0000 UTC m=+1413.109525411" watchObservedRunningTime="2026-10-31 18:45:09.880423912 +0000 UTC m=+1413.111662203"
Oct 31 18:45:10.369721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:10.369688    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:10.373994 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:10.373963    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 18:45:10.374364 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:10.374334    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 18:45:11.372910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:11.372869    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 18:45:11.373557 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:11.373523    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 18:45:12.376877 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:12.376837    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 18:45:12.377374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:12.376903    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 18:45:13.371891 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.371854    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 18:45:13.372183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.372148    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 18:45:13.629299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.629264    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"]
Oct 31 18:45:13.633906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.633879    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.708406 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.708360    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-podinfo\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.737526 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.710785    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a768f9-2b7c-43f0-8f36-f59966d129ae-istiod-ca-cert\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.896542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.896422    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-data\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.896805 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.896780    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-envoy\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.919630 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.919590    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-socket\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.919967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.919944    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-credential-socket\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.920154 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.920135    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-token\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.920296 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.920276    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n785\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-kube-api-access-8n785\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.924614 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.740221    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"]
Oct 31 18:45:13.924614 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.920539    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-certs\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:13.924614 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:13.741104    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c21e9e-openshift-default-dockercfg-ht7cr\""
Oct 31 18:45:14.022147 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022110    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-token\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.022390 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022189    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n785\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-kube-api-access-8n785\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.022390 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022313    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-certs\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.022390 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022380    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-podinfo\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.026586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022434    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a768f9-2b7c-43f0-8f36-f59966d129ae-istiod-ca-cert\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.026586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022488    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-data\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.026586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022533    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-envoy\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.026586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022571    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-socket\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.026586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.022718    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-credential-socket\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.026586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.023233    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-credential-socket\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.026586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.024856    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-certs\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.026586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.026125    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-socket\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.027059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.026971    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a768f9-2b7c-43f0-8f36-f59966d129ae-istiod-ca-cert\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.027248 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.027210    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-data\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.033038 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.033014    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-podinfo\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.034629 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.034588    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-envoy\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.087797 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.087765    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n785\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-kube-api-access-8n785\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.089329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.089294    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-token\") pod \"e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") " pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.249387 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.249287    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:14.384894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.384497    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 18:45:14.384894 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.384553    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 18:45:14.747978 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.747951    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"]
Oct 31 18:45:14.763480 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:45:14.756918    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a768f9_2b7c_43f0_8f36_f59966d129ae.slice/crio-ef140d4f2eaae60d3b0a4af0975647b897bf6189c88c409fa649e9f00c5afcd2 WatchSource:0}: Error finding container ef140d4f2eaae60d3b0a4af0975647b897bf6189c88c409fa649e9f00c5afcd2: Status 404 returned error can't find the container with id ef140d4f2eaae60d3b0a4af0975647b897bf6189c88c409fa649e9f00c5afcd2
Oct 31 18:45:14.780004 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.779957    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:45:14.780105 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.780054    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:45:14.780156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.780128    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:45:14.928010 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.927965    5551 scope.go:122] "RemoveContainer" containerID="6c6ec1b900edf2c16a57d653b7d33879d2b70ae2e9b2e31ab89c20343e64aa40"
Oct 31 18:45:14.928220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:14.928203    5551 scope.go:122] "RemoveContainer" containerID="820a0ddff9cb4fab40bd0bcfc00ef7e095762f0b68ce893b25f2dd71683f8893"
Oct 31 18:45:15.012105 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:15.012066    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" event={"ID":"95a768f9-2b7c-43f0-8f36-f59966d129ae","Type":"ContainerStarted","Data":"45f8d89440a9c0841b8e391e1ad5c6d6ab442d449431864c88f6d1b57ba57dce"}
Oct 31 18:45:15.012239 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:15.012137    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" event={"ID":"95a768f9-2b7c-43f0-8f36-f59966d129ae","Type":"ContainerStarted","Data":"ef140d4f2eaae60d3b0a4af0975647b897bf6189c88c409fa649e9f00c5afcd2"}
Oct 31 18:45:15.137157 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:15.137092    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podStartSLOduration=2.137073638 podStartE2EDuration="2.137073638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:15.133622084 +0000 UTC m=+1418.364860411" watchObservedRunningTime="2026-10-31 18:45:15.137073638 +0000 UTC m=+1418.368311929"
Oct 31 18:45:15.262707 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:15.262624    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:15.267005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:15.265587    5551 patch_prober.go:29] interesting pod/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 18:45:15.267005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:15.265695    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 18:45:15.384226 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:15.384190    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 18:45:15.384385 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:15.384258    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 18:45:16.113161 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:16.113130    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" event={"ID":"6fe060b2-fa68-43b5-b9b9-a2a283dc7626","Type":"ContainerStarted","Data":"8ff3a374f97f659f246698c7d6dda7581248e1af4e4b4f6d58473849e22c6a79"}
Oct 31 18:45:16.115959 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:16.115928    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" event={"ID":"e0d8c690-5fb3-4e72-aa60-840831e184bc","Type":"ContainerStarted","Data":"ae3e4c98c2b45248f738cef89f32891cff04ee44e038bbe02594e921dbbe0f47"}
Oct 31 18:45:16.267601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:16.267571    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:16.617222 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:16.617167    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:45:17.128613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.128581    5551 generic.go:356] "Generic (PLEG): container finished" podID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerID="ae3e4c98c2b45248f738cef89f32891cff04ee44e038bbe02594e921dbbe0f47" exitCode=60
Oct 31 18:45:17.129103 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.128655    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" event={"ID":"e0d8c690-5fb3-4e72-aa60-840831e184bc","Type":"ContainerDied","Data":"ae3e4c98c2b45248f738cef89f32891cff04ee44e038bbe02594e921dbbe0f47"}
Oct 31 18:45:17.129103 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.128708    5551 scope.go:122] "RemoveContainer" containerID="820a0ddff9cb4fab40bd0bcfc00ef7e095762f0b68ce893b25f2dd71683f8893"
Oct 31 18:45:17.129103 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.129044    5551 scope.go:122] "RemoveContainer" containerID="ae3e4c98c2b45248f738cef89f32891cff04ee44e038bbe02594e921dbbe0f47"
Oct 31 18:45:17.129448 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:17.129424    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh_odh-ai-gateway-infra(e0d8c690-5fb3-4e72-aa60-840831e184bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc"
Oct 31 18:45:17.131331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.131305    5551 generic.go:356] "Generic (PLEG): container finished" podID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerID="8ff3a374f97f659f246698c7d6dda7581248e1af4e4b4f6d58473849e22c6a79" exitCode=60
Oct 31 18:45:17.131432 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.131338    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" event={"ID":"6fe060b2-fa68-43b5-b9b9-a2a283dc7626","Type":"ContainerDied","Data":"8ff3a374f97f659f246698c7d6dda7581248e1af4e4b4f6d58473849e22c6a79"}
Oct 31 18:45:17.131786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.131757    5551 scope.go:122] "RemoveContainer" containerID="8ff3a374f97f659f246698c7d6dda7581248e1af4e4b4f6d58473849e22c6a79"
Oct 31 18:45:17.132000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.131757    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:17.132324 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:17.132074    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc_odh-ai-gateway-infra(6fe060b2-fa68-43b5-b9b9-a2a283dc7626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626"
Oct 31 18:45:17.133566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.133549    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:45:17.143588 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.143564    5551 scope.go:122] "RemoveContainer" containerID="6c6ec1b900edf2c16a57d653b7d33879d2b70ae2e9b2e31ab89c20343e64aa40"
Oct 31 18:45:17.236271 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.236238    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"]
Oct 31 18:45:17.354824 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.354791    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"]
Oct 31 18:45:17.371662 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.371607    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:45:17.371843 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:17.371680    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:45:18.278518 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.276707    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:18.278518 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.277500    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:18.283824 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.282472    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:45:18.433531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.433367    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:18.512582 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.512553    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:18.536881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.536840    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-maas-api-service-ca\") pod \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\" (UID: \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\") "
Oct 31 18:45:18.537022 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.536970    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d8c690-5fb3-4e72-aa60-840831e184bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8c690-5fb3-4e72-aa60-840831e184bc-maas-api-service-ca\") pod \"e0d8c690-5fb3-4e72-aa60-840831e184bc\" (UID: \"e0d8c690-5fb3-4e72-aa60-840831e184bc\") "
Oct 31 18:45:18.537079 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.537023    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-kube-api-access-2zxsd\" (UniqueName: \"kubernetes.io/projected/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-kube-api-access-2zxsd\") pod \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\" (UID: \"6fe060b2-fa68-43b5-b9b9-a2a283dc7626\") "
Oct 31 18:45:18.537163 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.537128    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-maas-api-service-ca" pod "6fe060b2-fa68-43b5-b9b9-a2a283dc7626" (UID: "6fe060b2-fa68-43b5-b9b9-a2a283dc7626"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:18.537163 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.537138    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d8c690-5fb3-4e72-aa60-840831e184bc-kube-api-access-r4ds9\" (UniqueName: \"kubernetes.io/projected/e0d8c690-5fb3-4e72-aa60-840831e184bc-kube-api-access-r4ds9\") pod \"e0d8c690-5fb3-4e72-aa60-840831e184bc\" (UID: \"e0d8c690-5fb3-4e72-aa60-840831e184bc\") "
Oct 31 18:45:18.537398 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.537378    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d8c690-5fb3-4e72-aa60-840831e184bc-maas-api-service-ca" pod "e0d8c690-5fb3-4e72-aa60-840831e184bc" (UID: "e0d8c690-5fb3-4e72-aa60-840831e184bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:18.537913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.537869    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8c690-5fb3-4e72-aa60-840831e184bc-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:18.537913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.537890    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:18.539333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.539313    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d8c690-5fb3-4e72-aa60-840831e184bc-kube-api-access-r4ds9" pod "e0d8c690-5fb3-4e72-aa60-840831e184bc" (UID: "e0d8c690-5fb3-4e72-aa60-840831e184bc"). InnerVolumeSpecName "kube-api-access-r4ds9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:18.539489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.539438    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-kube-api-access-2zxsd" pod "6fe060b2-fa68-43b5-b9b9-a2a283dc7626" (UID: "6fe060b2-fa68-43b5-b9b9-a2a283dc7626"). InnerVolumeSpecName "kube-api-access-2zxsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:18.640257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.640210    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zxsd\" (UniqueName: \"kubernetes.io/projected/6fe060b2-fa68-43b5-b9b9-a2a283dc7626-kube-api-access-2zxsd\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:18.640257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:18.640255    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4ds9\" (UniqueName: \"kubernetes.io/projected/e0d8c690-5fb3-4e72-aa60-840831e184bc-kube-api-access-r4ds9\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:19.154673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.154636    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"
Oct 31 18:45:19.154860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.154664    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh" event={"ID":"e0d8c690-5fb3-4e72-aa60-840831e184bc","Type":"ContainerDied","Data":"f8bef1886ea2808ed35bdb2e90b94da25f0004d63769415707efc2a41a223177"}
Oct 31 18:45:19.154860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.154742    5551 scope.go:122] "RemoveContainer" containerID="ae3e4c98c2b45248f738cef89f32891cff04ee44e038bbe02594e921dbbe0f47"
Oct 31 18:45:19.156430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.156390    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc" event={"ID":"6fe060b2-fa68-43b5-b9b9-a2a283dc7626","Type":"ContainerDied","Data":"c333c087e140de5bdb98c994cb268f7efffafdd814b0574fd2e33a89a26e996b"}
Oct 31 18:45:19.157396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.156829    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"
Oct 31 18:45:19.165185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.165165    5551 scope.go:122] "RemoveContainer" containerID="8ff3a374f97f659f246698c7d6dda7581248e1af4e4b4f6d58473849e22c6a79"
Oct 31 18:45:19.324003 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.323969    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"]
Oct 31 18:45:19.331046 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.331020    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891205-8zzmh"]
Oct 31 18:45:19.442166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.442078    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"]
Oct 31 18:45:19.445915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:19.445888    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891205-5lkdc"]
Oct 31 18:45:20.909727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:20.909697    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" path="/var/lib/kubelet/pods/6fe060b2-fa68-43b5-b9b9-a2a283dc7626/volumes"
Oct 31 18:45:20.910153 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:20.910138    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" path="/var/lib/kubelet/pods/e0d8c690-5fb3-4e72-aa60-840831e184bc/volumes"
Oct 31 18:45:27.312050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:27.312018    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"]
Oct 31 18:45:32.312757 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:32.312713    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9" podUID="ad6b5522-274e-41c6-a6b3-487b329f4aa2" containerName="payload-processing" containerID="cri-o://0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514" gracePeriod=25
Oct 31 18:45:35.429206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:35.429167    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9" podUID="ad6b5522-274e-41c6-a6b3-487b329f4aa2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:45:45.429726 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:45.429633    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9" podUID="ad6b5522-274e-41c6-a6b3-487b329f4aa2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:45:46.605380 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:46.605340    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:45:52.215414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.215381    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"]
Oct 31 18:45:52.333697 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.333662    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"]
Oct 31 18:45:52.611332 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.611300    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:45:52.663487 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.659811    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6b5522-274e-41c6-a6b3-487b329f4aa2-kube-api-access-8sv2t\" (UniqueName: \"kubernetes.io/projected/ad6b5522-274e-41c6-a6b3-487b329f4aa2-kube-api-access-8sv2t\") pod \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\" (UID: \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\") "
Oct 31 18:45:52.663487 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.660091    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6b5522-274e-41c6-a6b3-487b329f4aa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6b5522-274e-41c6-a6b3-487b329f4aa2-plugins-config-volume\") pod \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\" (UID: \"ad6b5522-274e-41c6-a6b3-487b329f4aa2\") "
Oct 31 18:45:52.663487 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.662937    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6b5522-274e-41c6-a6b3-487b329f4aa2-plugins-config-volume" pod "ad6b5522-274e-41c6-a6b3-487b329f4aa2" (UID: "ad6b5522-274e-41c6-a6b3-487b329f4aa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:52.667488 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.665994    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6b5522-274e-41c6-a6b3-487b329f4aa2-kube-api-access-8sv2t" pod "ad6b5522-274e-41c6-a6b3-487b329f4aa2" (UID: "ad6b5522-274e-41c6-a6b3-487b329f4aa2"). InnerVolumeSpecName "kube-api-access-8sv2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:52.688988 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.688950    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6b5522-274e-41c6-a6b3-487b329f4aa2-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:52.689132 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:52.688992    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sv2t\" (UniqueName: \"kubernetes.io/projected/ad6b5522-274e-41c6-a6b3-487b329f4aa2-kube-api-access-8sv2t\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:53.424836 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.424801    5551 generic.go:356] "Generic (PLEG): container finished" podID="ad6b5522-274e-41c6-a6b3-487b329f4aa2" containerID="0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514" exitCode=0
Oct 31 18:45:53.424836 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.424838    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9" event={"ID":"ad6b5522-274e-41c6-a6b3-487b329f4aa2","Type":"ContainerDied","Data":"0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514"}
Oct 31 18:45:53.425330 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.424844    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"
Oct 31 18:45:53.425330 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.424878    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9" event={"ID":"ad6b5522-274e-41c6-a6b3-487b329f4aa2","Type":"ContainerDied","Data":"11bdaf5a89ca861a96890c30f6445d62759815d20e15a6fea95542165e5d90f3"}
Oct 31 18:45:53.425330 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.424899    5551 scope.go:122] "RemoveContainer" containerID="0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514"
Oct 31 18:45:53.435210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.435189    5551 scope.go:122] "RemoveContainer" containerID="0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514"
Oct 31 18:45:53.435527 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:45:53.435508    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514\": container with ID starting with 0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514 not found: ID does not exist" containerID="0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514"
Oct 31 18:45:53.435579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.435538    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514"} err="failed to get container status \"0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514\": rpc error: code = NotFound desc = could not find container \"0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514\": container with ID starting with 0e807719cd135230e245cdb373bc7d9309ac9fb09300b42a9cae30fefbe7f514 not found: ID does not exist"
Oct 31 18:45:53.757668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.754521    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"]
Oct 31 18:45:53.768003 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:53.767966    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-k56z9"]
Oct 31 18:45:54.910042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:54.910012    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6b5522-274e-41c6-a6b3-487b329f4aa2" path="/var/lib/kubelet/pods/ad6b5522-274e-41c6-a6b3-487b329f4aa2/volumes"
Oct 31 18:45:57.215851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.215774    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk" podUID="056e6a86-6619-4821-96e2-d43e08a1dd37" containerName="payload-processing" containerID="cri-o://6d6d32a2e2ed406f11777ef7322f49d9df7aaadef48cea305243f0c2c3a6396c" gracePeriod=25
Oct 31 18:45:57.334597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.334561    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67" podUID="4acb6dcf-ec1e-4972-84f2-8372cd240632" containerName="payload-pre-processing" containerID="cri-o://123867a64a15c1c82cc4ae84dda69ac030b80dbef74718e61d4ce2bea963cd10" gracePeriod=25
Oct 31 18:45:57.458397 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.458360    5551 generic.go:356] "Generic (PLEG): container finished" podID="4acb6dcf-ec1e-4972-84f2-8372cd240632" containerID="123867a64a15c1c82cc4ae84dda69ac030b80dbef74718e61d4ce2bea963cd10" exitCode=0
Oct 31 18:45:57.458561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.458434    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67" event={"ID":"4acb6dcf-ec1e-4972-84f2-8372cd240632","Type":"ContainerDied","Data":"123867a64a15c1c82cc4ae84dda69ac030b80dbef74718e61d4ce2bea963cd10"}
Oct 31 18:45:57.460913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.460778    5551 generic.go:356] "Generic (PLEG): container finished" podID="056e6a86-6619-4821-96e2-d43e08a1dd37" containerID="6d6d32a2e2ed406f11777ef7322f49d9df7aaadef48cea305243f0c2c3a6396c" exitCode=0
Oct 31 18:45:57.460913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.460839    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk" event={"ID":"056e6a86-6619-4821-96e2-d43e08a1dd37","Type":"ContainerDied","Data":"6d6d32a2e2ed406f11777ef7322f49d9df7aaadef48cea305243f0c2c3a6396c"}
Oct 31 18:45:57.508200 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.508169    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:45:57.612151 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.612126    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:45:57.672195 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.672158    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056e6a86-6619-4821-96e2-d43e08a1dd37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056e6a86-6619-4821-96e2-d43e08a1dd37-plugins-config-volume\") pod \"056e6a86-6619-4821-96e2-d43e08a1dd37\" (UID: \"056e6a86-6619-4821-96e2-d43e08a1dd37\") "
Oct 31 18:45:57.672377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.672362    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056e6a86-6619-4821-96e2-d43e08a1dd37-kube-api-access-jxztz\" (UniqueName: \"kubernetes.io/projected/056e6a86-6619-4821-96e2-d43e08a1dd37-kube-api-access-jxztz\") pod \"056e6a86-6619-4821-96e2-d43e08a1dd37\" (UID: \"056e6a86-6619-4821-96e2-d43e08a1dd37\") "
Oct 31 18:45:57.672618 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.672584    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056e6a86-6619-4821-96e2-d43e08a1dd37-plugins-config-volume" pod "056e6a86-6619-4821-96e2-d43e08a1dd37" (UID: "056e6a86-6619-4821-96e2-d43e08a1dd37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:57.672896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.672876    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056e6a86-6619-4821-96e2-d43e08a1dd37-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:57.674583 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.674562    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056e6a86-6619-4821-96e2-d43e08a1dd37-kube-api-access-jxztz" pod "056e6a86-6619-4821-96e2-d43e08a1dd37" (UID: "056e6a86-6619-4821-96e2-d43e08a1dd37"). InnerVolumeSpecName "kube-api-access-jxztz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:57.773426 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.773335    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4acb6dcf-ec1e-4972-84f2-8372cd240632-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4acb6dcf-ec1e-4972-84f2-8372cd240632-plugins-config-volume\") pod \"4acb6dcf-ec1e-4972-84f2-8372cd240632\" (UID: \"4acb6dcf-ec1e-4972-84f2-8372cd240632\") "
Oct 31 18:45:57.773426 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.773408    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acb6dcf-ec1e-4972-84f2-8372cd240632-kube-api-access-snt6s\" (UniqueName: \"kubernetes.io/projected/4acb6dcf-ec1e-4972-84f2-8372cd240632-kube-api-access-snt6s\") pod \"4acb6dcf-ec1e-4972-84f2-8372cd240632\" (UID: \"4acb6dcf-ec1e-4972-84f2-8372cd240632\") "
Oct 31 18:45:57.773726 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.773701    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acb6dcf-ec1e-4972-84f2-8372cd240632-plugins-config-volume" pod "4acb6dcf-ec1e-4972-84f2-8372cd240632" (UID: "4acb6dcf-ec1e-4972-84f2-8372cd240632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:57.774265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.774240    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4acb6dcf-ec1e-4972-84f2-8372cd240632-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:57.774355 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.774271    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxztz\" (UniqueName: \"kubernetes.io/projected/056e6a86-6619-4821-96e2-d43e08a1dd37-kube-api-access-jxztz\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:57.775796 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.775765    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acb6dcf-ec1e-4972-84f2-8372cd240632-kube-api-access-snt6s" pod "4acb6dcf-ec1e-4972-84f2-8372cd240632" (UID: "4acb6dcf-ec1e-4972-84f2-8372cd240632"). InnerVolumeSpecName "kube-api-access-snt6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:57.877272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:57.876721    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snt6s\" (UniqueName: \"kubernetes.io/projected/4acb6dcf-ec1e-4972-84f2-8372cd240632-kube-api-access-snt6s\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:58.469414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.469384    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"
Oct 31 18:45:58.469893 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.469383    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk" event={"ID":"056e6a86-6619-4821-96e2-d43e08a1dd37","Type":"ContainerDied","Data":"0c5a48487be37b220e02355c089f4719ace6d8633e71d4d6fb0a08581bf00b75"}
Oct 31 18:45:58.469893 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.469543    5551 scope.go:122] "RemoveContainer" containerID="6d6d32a2e2ed406f11777ef7322f49d9df7aaadef48cea305243f0c2c3a6396c"
Oct 31 18:45:58.470948 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.470917    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"
Oct 31 18:45:58.471005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.470960    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67" event={"ID":"4acb6dcf-ec1e-4972-84f2-8372cd240632","Type":"ContainerDied","Data":"e47ffbbc29e207fb4bdaf4a22bd6760c6c766383b1cabac7da26b6301114bb49"}
Oct 31 18:45:58.480780 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.480678    5551 scope.go:122] "RemoveContainer" containerID="123867a64a15c1c82cc4ae84dda69ac030b80dbef74718e61d4ce2bea963cd10"
Oct 31 18:45:58.635101 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.635066    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"]
Oct 31 18:45:58.684331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.684294    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd6c0a-868fdc775f-s85zk"]
Oct 31 18:45:58.747408 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.747330    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"]
Oct 31 18:45:58.755396 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.755363    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd6c0a-75667b7d5f-7jh67"]
Oct 31 18:45:58.909405 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.909370    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056e6a86-6619-4821-96e2-d43e08a1dd37" path="/var/lib/kubelet/pods/056e6a86-6619-4821-96e2-d43e08a1dd37/volumes"
Oct 31 18:45:58.909795 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:45:58.909780    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4acb6dcf-ec1e-4972-84f2-8372cd240632" path="/var/lib/kubelet/pods/4acb6dcf-ec1e-4972-84f2-8372cd240632/volumes"
Oct 31 18:46:07.455903 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.455865    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"]
Oct 31 18:46:07.458041 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458007    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:46:07.458041 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458040    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458071    5551 cpu_manager.go:414] "containerMap: removing container" podUID="ad6b5522-274e-41c6-a6b3-487b329f4aa2" containerName="payload-processing"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458079    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6b5522-274e-41c6-a6b3-487b329f4aa2" containerName="payload-processing"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458132    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4acb6dcf-ec1e-4972-84f2-8372cd240632" containerName="payload-pre-processing"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458141    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4acb6dcf-ec1e-4972-84f2-8372cd240632" containerName="payload-pre-processing"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458157    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458163    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458187    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458195    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458231    5551 cpu_manager.go:414] "containerMap: removing container" podUID="056e6a86-6619-4821-96e2-d43e08a1dd37" containerName="payload-processing"
Oct 31 18:46:07.458234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458239    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056e6a86-6619-4821-96e2-d43e08a1dd37" containerName="payload-processing"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458250    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458257    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458287    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458293    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458572    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458596    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6b5522-274e-41c6-a6b3-487b329f4aa2" containerName="payload-processing"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458622    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4acb6dcf-ec1e-4972-84f2-8372cd240632" containerName="payload-pre-processing"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458637    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458648    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458665    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="056e6a86-6619-4821-96e2-d43e08a1dd37" containerName="payload-processing"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458680    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:46:07.458692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.458687    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:46:07.465191 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.465167    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.513252 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.513216    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"]
Oct 31 18:46:07.523102 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.523065    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/018b2948-32ed-4f73-8be5-e4a390a6079f-openshift-service-ca\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.523313 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.523288    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-metrics-certs\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.523447 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.523425    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5cg\" (UniqueName: \"kubernetes.io/projected/018b2948-32ed-4f73-8be5-e4a390a6079f-kube-api-access-6s5cg\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.523547 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.523529    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.558050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.558016    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c21e9e-serving-cert\""
Oct 31 18:46:07.624923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.624888    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/018b2948-32ed-4f73-8be5-e4a390a6079f-openshift-service-ca\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.625124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.625037    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-metrics-certs\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.625124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.625096    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5cg\" (UniqueName: \"kubernetes.io/projected/018b2948-32ed-4f73-8be5-e4a390a6079f-kube-api-access-6s5cg\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.625256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.625124    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.625446 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:46:07.625422    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-serving-cert: secret "maas-api-e2e-api-a-c21e9e-serving-cert" not found
Oct 31 18:46:07.625555 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:46:07.625517    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls podName:018b2948-32ed-4f73-8be5-e4a390a6079f nodeName:}" failed. No retries permitted until 2026-10-31 18:46:08.125491395 +0000 UTC m=+1471.356729680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls") pod "maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q" (UID: "018b2948-32ed-4f73-8be5-e4a390a6079f") : secret "maas-api-e2e-api-a-c21e9e-serving-cert" not found
Oct 31 18:46:07.626070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.626048    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/018b2948-32ed-4f73-8be5-e4a390a6079f-openshift-service-ca\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.628563 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.628527    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-metrics-certs\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:07.692231 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:07.692190    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5cg\" (UniqueName: \"kubernetes.io/projected/018b2948-32ed-4f73-8be5-e4a390a6079f-kube-api-access-6s5cg\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:08.133114 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:08.133076    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:08.135657 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:08.135632    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls\") pod \"maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:08.379574 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:08.379545    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:08.762428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:08.762403    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"]
Oct 31 18:46:08.763343 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:46:08.763314    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018b2948_32ed_4f73_8be5_e4a390a6079f.slice/crio-32dddc8338b21ba33ef4e73077fda2116e92dec875f9ef68d1034b107a3672c3 WatchSource:0}: Error finding container 32dddc8338b21ba33ef4e73077fda2116e92dec875f9ef68d1034b107a3672c3: Status 404 returned error can't find the container with id 32dddc8338b21ba33ef4e73077fda2116e92dec875f9ef68d1034b107a3672c3
Oct 31 18:46:09.557803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:09.557764    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q" event={"ID":"018b2948-32ed-4f73-8be5-e4a390a6079f","Type":"ContainerStarted","Data":"32dddc8338b21ba33ef4e73077fda2116e92dec875f9ef68d1034b107a3672c3"}
Oct 31 18:46:10.611484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:10.610666    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q" event={"ID":"018b2948-32ed-4f73-8be5-e4a390a6079f","Type":"ContainerStarted","Data":"7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63"}
Oct 31 18:46:10.611484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:10.610760    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:10.725878 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:10.725824    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q" podStartSLOduration=2.835986279 podStartE2EDuration="3.725809171s" totalImagesPullingTime="889.822892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:46:10.724196577 +0000 UTC m=+1473.955434868" watchObservedRunningTime="2026-10-31 18:46:10.725809171 +0000 UTC m=+1473.957047461"
Oct 31 18:46:16.607166 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:46:16.607113    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:46:16.622351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:16.622327    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:46:39.937028 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:39.936997    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:46:39.987444 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:39.987413    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:46:40.871959 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:40.871862    5551 generic.go:356] "Generic (PLEG): container finished" podID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerID="45f8d89440a9c0841b8e391e1ad5c6d6ab442d449431864c88f6d1b57ba57dce" exitCode=0
Oct 31 18:46:40.872142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:40.871947    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" event={"ID":"95a768f9-2b7c-43f0-8f36-f59966d129ae","Type":"ContainerDied","Data":"45f8d89440a9c0841b8e391e1ad5c6d6ab442d449431864c88f6d1b57ba57dce"}
Oct 31 18:46:40.872417 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:40.872404    5551 scope.go:122] "RemoveContainer" containerID="45f8d89440a9c0841b8e391e1ad5c6d6ab442d449431864c88f6d1b57ba57dce"
Oct 31 18:46:40.874086 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:40.874060    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:46:40.874159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:40.874128    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:46:40.874194 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:40.874173    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:46:41.892544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:41.892503    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" event={"ID":"95a768f9-2b7c-43f0-8f36-f59966d129ae","Type":"ContainerStarted","Data":"6a6de976ff5eec30f6f0b4febc087d537e45e7e038e54eb4f432d81e26d431d8"}
Oct 31 18:46:42.263672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:42.263579    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:46:42.264006 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:42.263918    5551 patch_prober.go:29] interesting pod/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 18:46:42.264158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:42.264136    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 18:46:43.267485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:43.263781    5551 patch_prober.go:29] interesting pod/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 18:46:43.267485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:43.263838    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 18:46:44.268481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:44.264800    5551 patch_prober.go:29] interesting pod/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 18:46:44.268481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:44.264860    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 18:46:45.263905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:45.263865    5551 patch_prober.go:29] interesting pod/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 18:46:45.264193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:45.264166    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 18:46:46.266482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:46.265778    5551 patch_prober.go:29] interesting pod/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 18:46:46.266482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:46.265849    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 18:46:46.610592 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:46:46.610508    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:46:47.132213 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:47.132169    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:46:47.264054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:47.264013    5551 patch_prober.go:29] interesting pod/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 18:46:47.264264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:47.264085    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 18:46:48.266495 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.266440    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:46:48.267496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.267447    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:46:48.516301 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.516264    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"]
Oct 31 18:46:48.516536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.516487    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2" podUID="caf76a82-fac2-44c6-bb38-514958e44c06" containerName="maas-api" containerID="cri-o://091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea" gracePeriod=30
Oct 31 18:46:48.842000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.841967    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:46:48.950559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.950525    5551 generic.go:356] "Generic (PLEG): container finished" podID="caf76a82-fac2-44c6-bb38-514958e44c06" containerID="091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea" exitCode=0
Oct 31 18:46:48.950849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.950570    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"
Oct 31 18:46:48.951000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.950588    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2" event={"ID":"caf76a82-fac2-44c6-bb38-514958e44c06","Type":"ContainerDied","Data":"091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea"}
Oct 31 18:46:48.951000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.950973    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2" event={"ID":"caf76a82-fac2-44c6-bb38-514958e44c06","Type":"ContainerDied","Data":"9cfe179002ce6cc7f297fb183b288be057fd14456e02d1792baacc5d14649afc"}
Oct 31 18:46:48.951118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.951093    5551 scope.go:122] "RemoveContainer" containerID="091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea"
Oct 31 18:46:48.952247 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.952221    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf76a82-fac2-44c6-bb38-514958e44c06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf76a82-fac2-44c6-bb38-514958e44c06-openshift-service-ca\") pod \"caf76a82-fac2-44c6-bb38-514958e44c06\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") "
Oct 31 18:46:48.952384 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.952336    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf76a82-fac2-44c6-bb38-514958e44c06-kube-api-access-h2kj6\" (UniqueName: \"kubernetes.io/projected/caf76a82-fac2-44c6-bb38-514958e44c06-kube-api-access-h2kj6\") pod \"caf76a82-fac2-44c6-bb38-514958e44c06\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") "
Oct 31 18:46:48.952480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.952427    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls\") pod \"caf76a82-fac2-44c6-bb38-514958e44c06\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") "
Oct 31 18:46:48.952640 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.952530    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-metrics-certs\") pod \"caf76a82-fac2-44c6-bb38-514958e44c06\" (UID: \"caf76a82-fac2-44c6-bb38-514958e44c06\") "
Oct 31 18:46:48.954162 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.952679    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf76a82-fac2-44c6-bb38-514958e44c06-openshift-service-ca" pod "caf76a82-fac2-44c6-bb38-514958e44c06" (UID: "caf76a82-fac2-44c6-bb38-514958e44c06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:46:48.954162 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.953107    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf76a82-fac2-44c6-bb38-514958e44c06-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:49.000335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:48.967020    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-metrics-certs" pod "caf76a82-fac2-44c6-bb38-514958e44c06" (UID: "caf76a82-fac2-44c6-bb38-514958e44c06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:46:49.023179 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.022525    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls" pod "caf76a82-fac2-44c6-bb38-514958e44c06" (UID: "caf76a82-fac2-44c6-bb38-514958e44c06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:46:49.025788 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.025750    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf76a82-fac2-44c6-bb38-514958e44c06-kube-api-access-h2kj6" pod "caf76a82-fac2-44c6-bb38-514958e44c06" (UID: "caf76a82-fac2-44c6-bb38-514958e44c06"). InnerVolumeSpecName "kube-api-access-h2kj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:46:49.027683 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.027661    5551 scope.go:122] "RemoveContainer" containerID="091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea"
Oct 31 18:46:49.028121 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:46:49.028093    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea\": container with ID starting with 091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea not found: ID does not exist" containerID="091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea"
Oct 31 18:46:49.028209 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.028136    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea"} err="failed to get container status \"091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea\": rpc error: code = NotFound desc = could not find container \"091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea\": container with ID starting with 091a51019af66294ea202dc560f5f21fa2f3a7d4c41028c6b329fc40a6b1daea not found: ID does not exist"
Oct 31 18:46:49.055379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.055340    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2kj6\" (UniqueName: \"kubernetes.io/projected/caf76a82-fac2-44c6-bb38-514958e44c06-kube-api-access-h2kj6\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:49.055379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.055370    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:49.055379 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.055380    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf76a82-fac2-44c6-bb38-514958e44c06-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:49.372642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.372609    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"]
Oct 31 18:46:49.377093 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:49.377065    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-2xrz2"]
Oct 31 18:46:50.911791 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:46:50.911755    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf76a82-fac2-44c6-bb38-514958e44c06" path="/var/lib/kubelet/pods/caf76a82-fac2-44c6-bb38-514958e44c06/volumes"
Oct 31 18:47:10.292837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:10.292795    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"]
Oct 31 18:47:10.293338 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:10.293017    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy" containerID="cri-o://bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c" gracePeriod=30
Oct 31 18:47:10.344279 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:10.344250    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"]
Oct 31 18:47:10.344561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:10.344533    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy" containerID="cri-o://bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb" gracePeriod=30
Oct 31 18:47:10.348654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:10.348626    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"]
Oct 31 18:47:10.348863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:10.348837    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy" containerID="cri-o://644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92" gracePeriod=30
Oct 31 18:47:10.392883 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:10.392856    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"]
Oct 31 18:47:10.393096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:10.393067    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy" containerID="cri-o://012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b" gracePeriod=30
Oct 31 18:47:15.679291 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.679252    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:47:15.742680 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.742648    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:47:15.786284 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.786258    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:47:15.789707 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.789687    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:47:15.838176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.837878    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-socket\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.838176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.837955    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-credential-socket\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838011    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-podinfo\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.838176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838085    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9f330e-d686-4475-8579-a0d779d0230c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9f330e-d686-4475-8579-a0d779d0230c-istiod-ca-cert\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838121    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-kube-api-access-6cn6t\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-kube-api-access-6cn6t\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838149    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-kube-api-access-pkjv7\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.838176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838182    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-certs\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838226    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-envoy\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838260    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-istio-token\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-istio-token\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838314    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-socket\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838310    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-socket" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838345    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f9f330e-d686-4475-8579-a0d779d0230c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9f330e-d686-4475-8579-a0d779d0230c-istio-podinfo\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838477    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-data\") pod \"5f9f330e-d686-4475-8579-a0d779d0230c\" (UID: \"5f9f330e-d686-4475-8579-a0d779d0230c\") "
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838515    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa63e23-0c15-440c-9e30-3bb576e312f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa63e23-0c15-440c-9e30-3bb576e312f1-istiod-ca-cert\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838619    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-data\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.838700 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838692    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-certs\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.839198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838728    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-envoy\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.839198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838772    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-token\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-token\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.839198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838804    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-certs" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.839198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.838812    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-credential-socket\") pod \"aaa63e23-0c15-440c-9e30-3bb576e312f1\" (UID: \"aaa63e23-0c15-440c-9e30-3bb576e312f1\") "
Oct 31 18:47:15.839198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.839026    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-credential-socket" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.839529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.839215    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-socket" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.840580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.840216    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.840580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.840246    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.840580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.840262    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.840580 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.840279    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.845602 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.842089    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-data" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.845602 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.842640    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa63e23-0c15-440c-9e30-3bb576e312f1-istiod-ca-cert" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:15.845602 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.845301    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f9f330e-d686-4475-8579-a0d779d0230c-istio-podinfo" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:15.845732 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.845636    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-data" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.845784 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.845767    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9f330e-d686-4475-8579-a0d779d0230c-istiod-ca-cert" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:15.845994 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.845968    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-credential-socket" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.846168 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.846068    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-istio-token" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:15.846354 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.846335    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-certs" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.847748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.847704    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-envoy" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.848305 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.848268    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-token" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:15.848514 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.848479    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-envoy" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.848635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.848618    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-kube-api-access-6cn6t" pod "5f9f330e-d686-4475-8579-a0d779d0230c" (UID: "5f9f330e-d686-4475-8579-a0d779d0230c"). InnerVolumeSpecName "kube-api-access-6cn6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:15.851121 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.851097    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-podinfo" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:15.852013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.851985    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-kube-api-access-pkjv7" pod "aaa63e23-0c15-440c-9e30-3bb576e312f1" (UID: "aaa63e23-0c15-440c-9e30-3bb576e312f1"). InnerVolumeSpecName "kube-api-access-pkjv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:15.941561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941524    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-credential-socket\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.941750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941600    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-kube-api-access-972gs\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-kube-api-access-972gs\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.941750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941640    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-data\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.941750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941669    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-socket\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.941750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941694    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-credential-socket\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.941750 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941732    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-certs\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941767    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-kube-api-access-ljlcx\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-kube-api-access-ljlcx\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941849    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-credential-socket" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941865    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-envoy\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941930    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istiod-ca-cert\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941928    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-data" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941965    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-token\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.941969    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-credential-socket" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942008    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-envoy\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.942050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942051    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-certs\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.942539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942066    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-certs" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.942539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942110    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istiod-ca-cert\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.942539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942191    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-data\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.942539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942226    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-podinfo\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.942539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942255    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-token\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-token\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.942539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942371    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-certs" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.942539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942379    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-podinfo\") pod \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\" (UID: \"c5ca5d99-8de5-497d-aae0-6e5a6536b739\") "
Oct 31 18:47:15.942539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942432    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-socket\") pod \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\" (UID: \"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e\") "
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942736    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-socket" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942749    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istiod-ca-cert" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.942952    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-socket" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943227    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-data" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943254    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943275    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943289    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943302    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943317    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943331    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9f330e-d686-4475-8579-a0d779d0230c-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943344    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943357    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cn6t\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-kube-api-access-6cn6t\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943371    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/aaa63e23-0c15-440c-9e30-3bb576e312f1-kube-api-access-pkjv7\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943384    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943396    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943408    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9f330e-d686-4475-8579-a0d779d0230c-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943421    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943435    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943449    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.943450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943484    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.945322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943498    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9f330e-d686-4475-8579-a0d779d0230c-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.945322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943512    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9f330e-d686-4475-8579-a0d779d0230c-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.945322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943526    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa63e23-0c15-440c-9e30-3bb576e312f1-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.945322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943529    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istiod-ca-cert" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:15.945322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943539    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.945322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943574    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa63e23-0c15-440c-9e30-3bb576e312f1-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.945322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.943593    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:15.946331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.946200    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-token" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:15.946331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.946280    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-kube-api-access-972gs" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "kube-api-access-972gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:15.946447 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.946334    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-podinfo" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:15.946447 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.946398    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-podinfo" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:15.946602 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.946578    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-token" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:15.946602 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.946578    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-envoy" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.946720 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.946686    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-envoy" pod "c5ca5d99-8de5-497d-aae0-6e5a6536b739" (UID: "c5ca5d99-8de5-497d-aae0-6e5a6536b739"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:15.947365 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:15.947343    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-kube-api-access-ljlcx" pod "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" (UID: "8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e"). InnerVolumeSpecName "kube-api-access-ljlcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:16.045590 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045558    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlcx\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-kube-api-access-ljlcx\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045590 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045593    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045609    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045627    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045641    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045657    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045672    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045688    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045702    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ca5d99-8de5-497d-aae0-6e5a6536b739-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.045782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.045717    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972gs\" (UniqueName: \"kubernetes.io/projected/c5ca5d99-8de5-497d-aae0-6e5a6536b739-kube-api-access-972gs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:16.248430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.248342    5551 generic.go:356] "Generic (PLEG): container finished" podID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerID="bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb" exitCode=0
Oct 31 18:47:16.248430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.248396    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"
Oct 31 18:47:16.248671 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.248431    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" event={"ID":"aaa63e23-0c15-440c-9e30-3bb576e312f1","Type":"ContainerDied","Data":"bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb"}
Oct 31 18:47:16.248671 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.248503    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m" event={"ID":"aaa63e23-0c15-440c-9e30-3bb576e312f1","Type":"ContainerDied","Data":"bbc6bc4ac36fda17d35c93979cebf466ec87eb10f44e8f8f6aa9c215aff7ba5c"}
Oct 31 18:47:16.248671 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.248535    5551 scope.go:122] "RemoveContainer" containerID="bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb"
Oct 31 18:47:16.250361 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.250320    5551 generic.go:356] "Generic (PLEG): container finished" podID="5f9f330e-d686-4475-8579-a0d779d0230c" containerID="bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c" exitCode=0
Oct 31 18:47:16.250524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.250361    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"
Oct 31 18:47:16.250524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.250402    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" event={"ID":"5f9f330e-d686-4475-8579-a0d779d0230c","Type":"ContainerDied","Data":"bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c"}
Oct 31 18:47:16.250524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.250493    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq" event={"ID":"5f9f330e-d686-4475-8579-a0d779d0230c","Type":"ContainerDied","Data":"e98bee01d1cb3fa0f5300d220a658e5dbdaecfba33bac2b3fc38ae49e07fb35f"}
Oct 31 18:47:16.253095 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.253073    5551 generic.go:356] "Generic (PLEG): container finished" podID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerID="012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b" exitCode=0
Oct 31 18:47:16.253216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.253122    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"
Oct 31 18:47:16.253216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.253136    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" event={"ID":"c5ca5d99-8de5-497d-aae0-6e5a6536b739","Type":"ContainerDied","Data":"012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b"}
Oct 31 18:47:16.253216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.253178    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t" event={"ID":"c5ca5d99-8de5-497d-aae0-6e5a6536b739","Type":"ContainerDied","Data":"ee9026668fd877b81e95937a7080e9f2a98d599c5208accd5284b52d70081fa6"}
Oct 31 18:47:16.254753 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.254731    5551 generic.go:356] "Generic (PLEG): container finished" podID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerID="644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92" exitCode=0
Oct 31 18:47:16.254852 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.254768    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"
Oct 31 18:47:16.254852 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.254802    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" event={"ID":"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e","Type":"ContainerDied","Data":"644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92"}
Oct 31 18:47:16.254852 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.254836    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw" event={"ID":"8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e","Type":"ContainerDied","Data":"f1f871911f9f9830c2ad08827e23d981627498c03e0849baf04bf8b30451c06c"}
Oct 31 18:47:16.269067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.269049    5551 scope.go:122] "RemoveContainer" containerID="bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb"
Oct 31 18:47:16.269369 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:16.269344    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb\": container with ID starting with bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb not found: ID does not exist" containerID="bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb"
Oct 31 18:47:16.269433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.269376    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb"} err="failed to get container status \"bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb\": rpc error: code = NotFound desc = could not find container \"bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb\": container with ID starting with bda14e066208e567f94df8f554fb5fce9a41d7b0ef68c33303c6dcbdf594bdcb not found: ID does not exist"
Oct 31 18:47:16.269433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.269393    5551 scope.go:122] "RemoveContainer" containerID="bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c"
Oct 31 18:47:16.281790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.281770    5551 scope.go:122] "RemoveContainer" containerID="bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c"
Oct 31 18:47:16.282040 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:16.282019    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c\": container with ID starting with bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c not found: ID does not exist" containerID="bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c"
Oct 31 18:47:16.282092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.282048    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c"} err="failed to get container status \"bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c\": rpc error: code = NotFound desc = could not find container \"bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c\": container with ID starting with bc9f9e5e031bc88e90fc71071d6a1fd9a47f12cc0157b8533ae43ce976c7c62c not found: ID does not exist"
Oct 31 18:47:16.282092 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.282063    5551 scope.go:122] "RemoveContainer" containerID="012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b"
Oct 31 18:47:16.291173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.291153    5551 scope.go:122] "RemoveContainer" containerID="c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d"
Oct 31 18:47:16.300045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.300026    5551 scope.go:122] "RemoveContainer" containerID="012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b"
Oct 31 18:47:16.300363 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:16.300346    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b\": container with ID starting with 012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b not found: ID does not exist" containerID="012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b"
Oct 31 18:47:16.300423 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.300370    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b"} err="failed to get container status \"012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b\": rpc error: code = NotFound desc = could not find container \"012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b\": container with ID starting with 012d5b1f5ad065cf453b0decdf3d726ec72e9bae6d22cf85122d3d5d41508c6b not found: ID does not exist"
Oct 31 18:47:16.300423 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.300389    5551 scope.go:122] "RemoveContainer" containerID="c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d"
Oct 31 18:47:16.300960 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:16.300940    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d\": container with ID starting with c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d not found: ID does not exist" containerID="c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d"
Oct 31 18:47:16.301047 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.300965    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d"} err="failed to get container status \"c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d\": rpc error: code = NotFound desc = could not find container \"c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d\": container with ID starting with c4f18e526996a50f37468d0e474ff1703aa90f04e584bc96bc57a2483660385d not found: ID does not exist"
Oct 31 18:47:16.301047 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.300979    5551 scope.go:122] "RemoveContainer" containerID="644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92"
Oct 31 18:47:16.314134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.314109    5551 scope.go:122] "RemoveContainer" containerID="644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92"
Oct 31 18:47:16.314428 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:16.314406    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92\": container with ID starting with 644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92 not found: ID does not exist" containerID="644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92"
Oct 31 18:47:16.314586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.314436    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92"} err="failed to get container status \"644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92\": rpc error: code = NotFound desc = could not find container \"644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92\": container with ID starting with 644466b8717da208497b6be12c9cc79953f408473ee60fd1b6a48e931ac96b92 not found: ID does not exist"
Oct 31 18:47:16.500547 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.488529    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"]
Oct 31 18:47:16.507544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.506409    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-w4d8t"]
Oct 31 18:47:16.545283 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.545244    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"]
Oct 31 18:47:16.550352 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.550162    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-cts7m"]
Oct 31 18:47:16.614117 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:16.614072    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:47:16.695572 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.695529    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"]
Oct 31 18:47:16.727481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.724646    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-gb5pw"]
Oct 31 18:47:16.803531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.802731    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"]
Oct 31 18:47:16.811397 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.809070    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-dgvsq"]
Oct 31 18:47:16.910134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.910100    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" path="/var/lib/kubelet/pods/5f9f330e-d686-4475-8579-a0d779d0230c/volumes"
Oct 31 18:47:16.910626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.910612    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" path="/var/lib/kubelet/pods/8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e/volumes"
Oct 31 18:47:16.911025 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.911013    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" path="/var/lib/kubelet/pods/aaa63e23-0c15-440c-9e30-3bb576e312f1/volumes"
Oct 31 18:47:16.911774 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:16.911759    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" path="/var/lib/kubelet/pods/c5ca5d99-8de5-497d-aae0-6e5a6536b739/volumes"
Oct 31 18:47:24.000773 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:23.999246    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"]
Oct 31 18:47:24.000773 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:23.999537    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy" containerID="cri-o://52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6" gracePeriod=30
Oct 31 18:47:24.051546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:24.049988    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"]
Oct 31 18:47:24.051546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:24.050213    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy" containerID="cri-o://4d398d7e8fd431aa2e136f169d0ce37d85c1b409c43b069d0c8cc75573e87431" gracePeriod=30
Oct 31 18:47:29.353856 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.353828    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:47:29.378645 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.378614    5551 generic.go:356] "Generic (PLEG): container finished" podID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerID="52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6" exitCode=0
Oct 31 18:47:29.378775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.378731    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"
Oct 31 18:47:29.378775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.378739    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" event={"ID":"59b38d11-b6ac-4102-8b62-d06f4ef93366","Type":"ContainerDied","Data":"52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6"}
Oct 31 18:47:29.378891 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.378799    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc" event={"ID":"59b38d11-b6ac-4102-8b62-d06f4ef93366","Type":"ContainerDied","Data":"8b40552c0cdb4af04adde6a1db3b48327e9f7239fc06a92a5c2a1df5a7a21ca7"}
Oct 31 18:47:29.378891 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.378826    5551 scope.go:122] "RemoveContainer" containerID="52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6"
Oct 31 18:47:29.380584 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.380547    5551 generic.go:356] "Generic (PLEG): container finished" podID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerID="4d398d7e8fd431aa2e136f169d0ce37d85c1b409c43b069d0c8cc75573e87431" exitCode=0
Oct 31 18:47:29.380702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.380612    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" event={"ID":"76f1579e-c88a-45f9-8130-ddb44b5ace3e","Type":"ContainerDied","Data":"4d398d7e8fd431aa2e136f169d0ce37d85c1b409c43b069d0c8cc75573e87431"}
Oct 31 18:47:29.380702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.380650    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf" event={"ID":"76f1579e-c88a-45f9-8130-ddb44b5ace3e","Type":"ContainerDied","Data":"573bbf09e438b44bc385f14a3413447abfdf97c326932313d676441df073234d"}
Oct 31 18:47:29.380702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.380674    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="573bbf09e438b44bc385f14a3413447abfdf97c326932313d676441df073234d"
Oct 31 18:47:29.383828 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.383813    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:47:29.390298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.390276    5551 scope.go:122] "RemoveContainer" containerID="52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6"
Oct 31 18:47:29.390611 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:29.390591    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6\": container with ID starting with 52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6 not found: ID does not exist" containerID="52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6"
Oct 31 18:47:29.390681 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.390618    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6"} err="failed to get container status \"52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6\": rpc error: code = NotFound desc = could not find container \"52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6\": container with ID starting with 52be42d689025841a5fe2808691db83ef1d4ed4958630c953363083da37ff3d6 not found: ID does not exist"
Oct 31 18:47:29.445554 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445515    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-token\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-token\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.445675 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445590    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-credential-socket\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.445675 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445643    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b38d11-b6ac-4102-8b62-d06f4ef93366-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b38d11-b6ac-4102-8b62-d06f4ef93366-istiod-ca-cert\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.445746 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445707    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-socket\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.445778 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445762    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-certs\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.445850 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445833    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-podinfo\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.445910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445875    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-certs\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.445961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445933    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-socket" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.445961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445904    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-credential-socket" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.446075 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.445983    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istiod-ca-cert\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.446075 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446039    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-socket\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.446181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446085    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-data\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.446181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446094    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-certs" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.446181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446113    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-podinfo\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.446181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446115    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b38d11-b6ac-4102-8b62-d06f4ef93366-istiod-ca-cert" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:29.446181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446126    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-certs" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.446181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446147    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-credential-socket\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.446496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446209    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-token\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-token\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.446496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446236    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-kube-api-access-lxr7g\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-kube-api-access-lxr7g\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.446496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446293    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-envoy\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.446496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446307    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-socket" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.446496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446323    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-envoy\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.446496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446358    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-data" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.446496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446379    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istiod-ca-cert" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:29.446496 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446386    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-data\") pod \"59b38d11-b6ac-4102-8b62-d06f4ef93366\" (UID: \"59b38d11-b6ac-4102-8b62-d06f4ef93366\") "
Oct 31 18:47:29.446878 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446608    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-data" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.446878 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.446535    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-kube-api-access-656b4\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-kube-api-access-656b4\") pod \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\" (UID: \"76f1579e-c88a-45f9-8130-ddb44b5ace3e\") "
Oct 31 18:47:29.447325 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447189    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-credential-socket" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.447492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447380    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.447492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447399    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.447492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447415    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.447492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447429    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.447492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447445    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.447492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447478    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.447492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447492    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.448020 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447505    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b38d11-b6ac-4102-8b62-d06f4ef93366-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.448020 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447519    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.448020 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.447532    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.449166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.449062    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-podinfo" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:29.449166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.449119    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-token" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:29.449166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.449103    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-podinfo" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:29.449334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.449141    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-envoy" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.449616 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.449589    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-kube-api-access-656b4" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "kube-api-access-656b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:29.449616 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.449593    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-envoy" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:29.449765 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.449695    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-token" pod "76f1579e-c88a-45f9-8130-ddb44b5ace3e" (UID: "76f1579e-c88a-45f9-8130-ddb44b5ace3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:29.450612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.450590    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-kube-api-access-lxr7g" pod "59b38d11-b6ac-4102-8b62-d06f4ef93366" (UID: "59b38d11-b6ac-4102-8b62-d06f4ef93366"). InnerVolumeSpecName "kube-api-access-lxr7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:29.549070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.549037    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.549070 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.549067    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.549310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.549080    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.549310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.549089    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxr7g\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-kube-api-access-lxr7g\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.549310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.549099    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f1579e-c88a-45f9-8130-ddb44b5ace3e-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.549310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.549107    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.549310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.549115    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-656b4\" (UniqueName: \"kubernetes.io/projected/76f1579e-c88a-45f9-8130-ddb44b5ace3e-kube-api-access-656b4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.549310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.549123    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b38d11-b6ac-4102-8b62-d06f4ef93366-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:29.817370 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.817326    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"]
Oct 31 18:47:29.825139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:29.824969    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-bjcpc"]
Oct 31 18:47:30.390086 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:30.390058    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"
Oct 31 18:47:30.558688 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:30.558616    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"]
Oct 31 18:47:30.606923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:30.606896    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-55kmf"]
Oct 31 18:47:30.909343 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:30.909306    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" path="/var/lib/kubelet/pods/59b38d11-b6ac-4102-8b62-d06f4ef93366/volumes"
Oct 31 18:47:30.909867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:30.909852    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" path="/var/lib/kubelet/pods/76f1579e-c88a-45f9-8130-ddb44b5ace3e/volumes"
Oct 31 18:47:32.788768 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.788734    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"]
Oct 31 18:47:32.789417 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789398    5551 cpu_manager.go:414] "containerMap: removing container" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy"
Oct 31 18:47:32.789477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789420    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy"
Oct 31 18:47:32.789477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789432    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy"
Oct 31 18:47:32.789477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789438    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy"
Oct 31 18:47:32.789477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789450    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy"
Oct 31 18:47:32.789477 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789467    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789483    5551 cpu_manager.go:414] "containerMap: removing container" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789488    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789495    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789500    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe060b2-fa68-43b5-b9b9-a2a283dc7626" containerName="cleanup"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789511    5551 cpu_manager.go:414] "containerMap: removing container" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789516    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789527    5551 cpu_manager.go:414] "containerMap: removing container" podUID="caf76a82-fac2-44c6-bb38-514958e44c06" containerName="maas-api"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789534    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf76a82-fac2-44c6-bb38-514958e44c06" containerName="maas-api"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789546    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy"
Oct 31 18:47:32.789637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789553    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789699    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f1579e-c88a-45f9-8130-ddb44b5ace3e" containerName="istio-proxy"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789707    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf76a82-fac2-44c6-bb38-514958e44c06" containerName="maas-api"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789715    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9f330e-d686-4475-8579-a0d779d0230c" containerName="istio-proxy"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789722    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d8c690-5fb3-4e72-aa60-840831e184bc" containerName="cleanup"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789729    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789738    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa63e23-0c15-440c-9e30-3bb576e312f1" containerName="istio-proxy"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789743    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1a1aa0-84fb-4608-9b4f-f3ac3a1b304e" containerName="istio-proxy"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789749    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy"
Oct 31 18:47:32.789945 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.789755    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b38d11-b6ac-4102-8b62-d06f4ef93366" containerName="istio-proxy"
Oct 31 18:47:32.793067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.793050    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.852647 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.852618    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"]
Oct 31 18:47:32.896594 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.896540    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5d1b3f-openshift-default-dockercfg-nnv5x\""
Oct 31 18:47:32.897155 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897126    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-credential-socket\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.897293 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897248    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-envoy\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.897345 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897300    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-token\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.897397 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897389    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8g5r\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-kube-api-access-f8g5r\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.897525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897497    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-podinfo\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.897968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897540    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-socket\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.897968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897609    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6038f7c4-9cd7-445a-bde8-952014f798fb-istiod-ca-cert\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.897968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897628    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-data\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:32.897968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:32.897676    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-certs\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.019485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019378    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-token\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.019673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019530    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8g5r\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-kube-api-access-f8g5r\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.019673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019597    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-podinfo\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.019673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019646    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-socket\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.019861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019706    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6038f7c4-9cd7-445a-bde8-952014f798fb-istiod-ca-cert\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.019861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019730    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-data\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.019861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019806    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-certs\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.019861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019855    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-credential-socket\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.020060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.019936    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-envoy\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.029486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.021557    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-data\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.029486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.022023    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-socket\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.029486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.022266    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-certs\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.029486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.022369    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6038f7c4-9cd7-445a-bde8-952014f798fb-istiod-ca-cert\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.029486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.022622    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-credential-socket\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.029908 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.029855    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-envoy\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.065527 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.064796    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-podinfo\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.086942 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.086910    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8g5r\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-kube-api-access-f8g5r\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.092080 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.092025    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-token\") pod \"e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") " pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.107797 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.107755    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:33.562562 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.562539    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"]
Oct 31 18:47:33.563406 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:47:33.563379    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6038f7c4_9cd7_445a_bde8_952014f798fb.slice/crio-8857e9dafd46b8e23aaadb039297f3971e31ec1ed0ea3d94275f933d926e7fdd WatchSource:0}: Error finding container 8857e9dafd46b8e23aaadb039297f3971e31ec1ed0ea3d94275f933d926e7fdd: Status 404 returned error can't find the container with id 8857e9dafd46b8e23aaadb039297f3971e31ec1ed0ea3d94275f933d926e7fdd
Oct 31 18:47:33.565805 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.565774    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:47:33.565887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.565834    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:47:33.565887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:33.565865    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:47:34.294270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.294241    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"]
Oct 31 18:47:34.294877 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.294862    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy"
Oct 31 18:47:34.294915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.294878    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ca5d99-8de5-497d-aae0-6e5a6536b739" containerName="istio-proxy"
Oct 31 18:47:34.297143 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.297127    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.350993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.350954    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3417b822-serving-cert\""
Oct 31 18:47:34.402952 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.402927    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"]
Oct 31 18:47:34.419631 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.419602    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" event={"ID":"6038f7c4-9cd7-445a-bde8-952014f798fb","Type":"ContainerStarted","Data":"1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2"}
Oct 31 18:47:34.419745 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.419647    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" event={"ID":"6038f7c4-9cd7-445a-bde8-952014f798fb","Type":"ContainerStarted","Data":"8857e9dafd46b8e23aaadb039297f3971e31ec1ed0ea3d94275f933d926e7fdd"}
Oct 31 18:47:34.438336 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.438307    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac7383d-186a-48eb-923e-0faea300f803-openshift-service-ca\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.438502 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.438355    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tq7\" (UniqueName: \"kubernetes.io/projected/8ac7383d-186a-48eb-923e-0faea300f803-kube-api-access-r2tq7\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.438572 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.438490    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-metrics-certs\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.438733 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.438717    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.566545 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.566223    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.566545 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.566289    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac7383d-186a-48eb-923e-0faea300f803-openshift-service-ca\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.566545 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.566320    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tq7\" (UniqueName: \"kubernetes.io/projected/8ac7383d-186a-48eb-923e-0faea300f803-kube-api-access-r2tq7\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.566545 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.566418    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-metrics-certs\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.566545 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:34.566476    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-serving-cert: secret "maas-api-e2e-mt-3417b822-serving-cert" not found
Oct 31 18:47:34.566908 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:34.566563    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls podName:8ac7383d-186a-48eb-923e-0faea300f803 nodeName:}" failed. No retries permitted until 2026-10-31 18:47:35.066537778 +0000 UTC m=+1558.297776056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls") pod "maas-api-e2e-mt-3417b822-764cd85fdd-xt29p" (UID: "8ac7383d-186a-48eb-923e-0faea300f803") : secret "maas-api-e2e-mt-3417b822-serving-cert" not found
Oct 31 18:47:34.568255 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.567844    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac7383d-186a-48eb-923e-0faea300f803-openshift-service-ca\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.570672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.570649    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-metrics-certs\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:34.625315 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:34.625277    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tq7\" (UniqueName: \"kubernetes.io/projected/8ac7383d-186a-48eb-923e-0faea300f803-kube-api-access-r2tq7\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:35.074238 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.074206    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:35.076780 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.076757    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls\") pod \"maas-api-e2e-mt-3417b822-764cd85fdd-xt29p\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:35.108168 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.108145    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:35.113645 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.113623    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:35.211173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.211149    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:35.237240 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.237161    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" podStartSLOduration=3.237141076 podStartE2EDuration="3.237141076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:34.550531257 +0000 UTC m=+1557.781769576" watchObservedRunningTime="2026-10-31 18:47:35.237141076 +0000 UTC m=+1558.468379374"
Oct 31 18:47:35.437288 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.437258    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:35.438299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.438280    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:47:35.600430 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.600392    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"]
Oct 31 18:47:35.641759 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:47:35.641728    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac7383d_186a_48eb_923e_0faea300f803.slice/crio-3712b5f5c7e015f93a1889b829cf310172f05e4d87598918b70546a2976d4185 WatchSource:0}: Error finding container 3712b5f5c7e015f93a1889b829cf310172f05e4d87598918b70546a2976d4185: Status 404 returned error can't find the container with id 3712b5f5c7e015f93a1889b829cf310172f05e4d87598918b70546a2976d4185
Oct 31 18:47:35.643360 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:35.643342    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:47:36.444850 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:36.444818    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p" event={"ID":"8ac7383d-186a-48eb-923e-0faea300f803","Type":"ContainerStarted","Data":"3712b5f5c7e015f93a1889b829cf310172f05e4d87598918b70546a2976d4185"}
Oct 31 18:47:37.453808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:37.453765    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p" event={"ID":"8ac7383d-186a-48eb-923e-0faea300f803","Type":"ContainerStarted","Data":"d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc"}
Oct 31 18:47:37.454221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:37.453999    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:37.590073 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:37.584431    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p" podStartSLOduration=2.6049888599999997 podStartE2EDuration="3.584409915s" totalImagesPullingTime="979.421055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:37.581899429 +0000 UTC m=+1560.813137719" watchObservedRunningTime="2026-10-31 18:47:37.584409915 +0000 UTC m=+1560.815648205"
Oct 31 18:47:38.742045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:38.742014    5551 scope.go:122] "RemoveContainer" containerID="4d398d7e8fd431aa2e136f169d0ce37d85c1b409c43b069d0c8cc75573e87431"
Oct 31 18:47:43.466024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:47:43.465996    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:47:46.605292 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:47:46.605257    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:48:01.190492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:01.190414    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"]
Oct 31 18:48:01.191001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:01.190716    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy" containerID="cri-o://515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9" gracePeriod=30
Oct 31 18:48:01.239282 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:01.239246    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"]
Oct 31 18:48:01.239577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:01.239530    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy" containerID="cri-o://7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e" gracePeriod=30
Oct 31 18:48:06.538837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.538803    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:48:06.559314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.559291    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:48:06.664066 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664036    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-kube-api-access-srfj4\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-kube-api-access-srfj4\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.664220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664096    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-envoy\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.664220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664155    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-data\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.664220 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664214    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-envoy\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.664394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664267    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-credential-socket\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.664394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664311    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-certs\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.664394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664389    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istiod-ca-cert\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.664635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664427    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-credential-socket\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.664635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664472    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-certs\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.664635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664508    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-data" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.664635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664514    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80db6ed8-783b-4782-af33-9e81981246f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80db6ed8-783b-4782-af33-9e81981246f4-istio-podinfo\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.664635 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664590    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-kube-api-access-t2l77\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-kube-api-access-t2l77\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.664927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664672    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-data\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.664927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664689    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-certs" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.664927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664738    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80db6ed8-783b-4782-af33-9e81981246f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80db6ed8-783b-4782-af33-9e81981246f4-istiod-ca-cert\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.664927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664771    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-socket\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.664927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664823    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-socket\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.664927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664844    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-credential-socket" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.664927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664892    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-token\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-token\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.664927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664896    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-credential-socket" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.665351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.664970    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-istio-token\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-istio-token\") pod \"80db6ed8-783b-4782-af33-9e81981246f4\" (UID: \"80db6ed8-783b-4782-af33-9e81981246f4\") "
Oct 31 18:48:06.665351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665007    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-podinfo\") pod \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\" (UID: \"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee\") "
Oct 31 18:48:06.665351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665170    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istiod-ca-cert" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:48:06.665351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665266    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-certs" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.665596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665413    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-socket" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.665660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665631    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-data" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.665755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665738    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.665822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665761    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.665822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665776    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.665822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665789    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.665822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665803    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.665822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665817    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.666264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665830    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.666264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665844    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.666264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.665904    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80db6ed8-783b-4782-af33-9e81981246f4-istiod-ca-cert" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:48:06.666264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.666140    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-socket" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.667472 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.667421    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80db6ed8-783b-4782-af33-9e81981246f4-istio-podinfo" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:48:06.667992 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.667952    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-podinfo" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:48:06.668392 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.668365    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-istio-token" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:06.668583 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.668558    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-envoy" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.668679 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.668618    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-envoy" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:06.668804 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.668788    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-kube-api-access-srfj4" pod "80db6ed8-783b-4782-af33-9e81981246f4" (UID: "80db6ed8-783b-4782-af33-9e81981246f4"). InnerVolumeSpecName "kube-api-access-srfj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:06.669236 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.669213    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-kube-api-access-t2l77" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "kube-api-access-t2l77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:06.670247 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.670223    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-token" pod "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" (UID: "4bd4c46f-43d9-4b17-bbcc-cb76561af9ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:06.718813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.718778    5551 generic.go:356] "Generic (PLEG): container finished" podID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerID="515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9" exitCode=0
Oct 31 18:48:06.718975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.718815    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" event={"ID":"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee","Type":"ContainerDied","Data":"515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9"}
Oct 31 18:48:06.718975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.718889    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72" event={"ID":"4bd4c46f-43d9-4b17-bbcc-cb76561af9ee","Type":"ContainerDied","Data":"732a9e9de13dcb6c29a38ce2ffc5da80dfb4db1c7770f8e75da7249c32f986e9"}
Oct 31 18:48:06.718975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.718830    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"
Oct 31 18:48:06.718975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.718924    5551 scope.go:122] "RemoveContainer" containerID="515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9"
Oct 31 18:48:06.720983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.720949    5551 generic.go:356] "Generic (PLEG): container finished" podID="80db6ed8-783b-4782-af33-9e81981246f4" containerID="7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e" exitCode=0
Oct 31 18:48:06.721107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.721016    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"
Oct 31 18:48:06.721107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.721086    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" event={"ID":"80db6ed8-783b-4782-af33-9e81981246f4","Type":"ContainerDied","Data":"7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e"}
Oct 31 18:48:06.721234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.721132    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4" event={"ID":"80db6ed8-783b-4782-af33-9e81981246f4","Type":"ContainerDied","Data":"49f43258e7960edbe6486d592f1371002f6e9f44a7009fb25e4cd9060718e007"}
Oct 31 18:48:06.733264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.733248    5551 scope.go:122] "RemoveContainer" containerID="515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9"
Oct 31 18:48:06.733584 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:48:06.733563    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9\": container with ID starting with 515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9 not found: ID does not exist" containerID="515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9"
Oct 31 18:48:06.733654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.733592    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9"} err="failed to get container status \"515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9\": rpc error: code = NotFound desc = could not find container \"515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9\": container with ID starting with 515ff91adc7022f96a002825d8b9da1ba645fddb0ec81d4ec6bdfa5234ebcca9 not found: ID does not exist"
Oct 31 18:48:06.733654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.733611    5551 scope.go:122] "RemoveContainer" containerID="7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e"
Oct 31 18:48:06.742996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.742977    5551 scope.go:122] "RemoveContainer" containerID="7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e"
Oct 31 18:48:06.743348 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:48:06.743292    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e\": container with ID starting with 7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e not found: ID does not exist" containerID="7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e"
Oct 31 18:48:06.743348 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.743315    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e"} err="failed to get container status \"7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e\": rpc error: code = NotFound desc = could not find container \"7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e\": container with ID starting with 7c42aaf70ad16c3295e341c0f34be7c0f4593cb864886f54b764d77716c0286e not found: ID does not exist"
Oct 31 18:48:06.767669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767615    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767640    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80db6ed8-783b-4782-af33-9e81981246f4-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767649    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2l77\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-kube-api-access-t2l77\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767658    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80db6ed8-783b-4782-af33-9e81981246f4-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767667    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80db6ed8-783b-4782-af33-9e81981246f4-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767677    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767684    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767693    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767704    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srfj4\" (UniqueName: \"kubernetes.io/projected/80db6ed8-783b-4782-af33-9e81981246f4-kube-api-access-srfj4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.767882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.767712    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:06.939670 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.939636    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"]
Oct 31 18:48:06.945686 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:06.945650    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dgjr4"]
Oct 31 18:48:07.070531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:07.070216    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"]
Oct 31 18:48:07.076503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:07.076451    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-w2p72"]
Oct 31 18:48:08.908979 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:08.908946    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" path="/var/lib/kubelet/pods/4bd4c46f-43d9-4b17-bbcc-cb76561af9ee/volumes"
Oct 31 18:48:08.909503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:08.909487    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80db6ed8-783b-4782-af33-9e81981246f4" path="/var/lib/kubelet/pods/80db6ed8-783b-4782-af33-9e81981246f4/volumes"
Oct 31 18:48:09.346934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:09.346839    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"]
Oct 31 18:48:09.347118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:09.347074    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" containerID="cri-o://f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168" gracePeriod=30
Oct 31 18:48:13.977960 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:13.977922    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:48:13.978407 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:13.977993    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:48:14.619870 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.619844    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:48:14.778755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.778667    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-token\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.778755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.778702    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-kube-api-access-xrbqm\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-kube-api-access-xrbqm\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.778755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.778740    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-certs\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.779037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.778810    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-envoy\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.779037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.778839    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-podinfo\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.779037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.778863    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istiod-ca-cert\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.779037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.778950    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-socket\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.779037 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779019    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-data\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.779301 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779068    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-credential-socket\") pod \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\" (UID: \"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da\") "
Oct 31 18:48:14.779358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779312    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-certs" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:14.779439 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779412    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istiod-ca-cert" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:48:14.779638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779492    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-socket" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:14.779638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779533    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-credential-socket" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:14.779638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779663    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-data" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:14.779958 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779939    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.779958 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779960    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.780126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779973    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.780126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779984    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.780126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.779997    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.781385 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.781364    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-kube-api-access-xrbqm" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "kube-api-access-xrbqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:14.781752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.781730    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-envoy" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:14.781848 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.781764    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-podinfo" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:48:14.781848 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.781767    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-token" pod "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" (UID: "8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:14.804207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.804180    5551 generic.go:356] "Generic (PLEG): container finished" podID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerID="f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168" exitCode=0
Oct 31 18:48:14.804345 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.804248    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" event={"ID":"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da","Type":"ContainerDied","Data":"f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168"}
Oct 31 18:48:14.804345 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.804301    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688" event={"ID":"8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da","Type":"ContainerDied","Data":"524d928578c46acf130c5fa1fa388b3b06c082612f5d30ca8da032b6dccb3b1e"}
Oct 31 18:48:14.804345 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.804331    5551 scope.go:122] "RemoveContainer" containerID="f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168"
Oct 31 18:48:14.804547 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.804531    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"
Oct 31 18:48:14.818979 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.818956    5551 scope.go:122] "RemoveContainer" containerID="f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168"
Oct 31 18:48:14.819324 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:48:14.819301    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168\": container with ID starting with f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168 not found: ID does not exist" containerID="f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168"
Oct 31 18:48:14.819402 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.819334    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168"} err="failed to get container status \"f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168\": rpc error: code = NotFound desc = could not find container \"f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168\": container with ID starting with f97e7439f3661421898769b035f075208eaa67a48f0aaebdae2eceaf3d53d168 not found: ID does not exist"
Oct 31 18:48:14.880778 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.880751    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.880778 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.880778    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrbqm\" (UniqueName: \"kubernetes.io/projected/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-kube-api-access-xrbqm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.880928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.880794    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.880928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.880805    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:14.973714 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.973678    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"]
Oct 31 18:48:14.976906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:14.976880    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-h7688"]
Oct 31 18:48:16.605482 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:48:16.605434    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:48:16.909512 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:16.909481    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" path="/var/lib/kubelet/pods/8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da/volumes"
Oct 31 18:48:17.647114 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647080    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"]
Oct 31 18:48:17.647642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647627    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy"
Oct 31 18:48:17.647691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647642    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy"
Oct 31 18:48:17.647691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647655    5551 cpu_manager.go:414] "containerMap: removing container" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy"
Oct 31 18:48:17.647691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647660    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy"
Oct 31 18:48:17.647691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647680    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy"
Oct 31 18:48:17.647691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647685    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy"
Oct 31 18:48:17.647887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647787    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbe7fbe-5aff-41b8-92f2-bef6bf61e7da" containerName="istio-proxy"
Oct 31 18:48:17.647887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647794    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="80db6ed8-783b-4782-af33-9e81981246f4" containerName="istio-proxy"
Oct 31 18:48:17.647887 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.647803    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd4c46f-43d9-4b17-bbcc-cb76561af9ee" containerName="istio-proxy"
Oct 31 18:48:17.652645 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.652626    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.715775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.715740    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.715928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.715780    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.715928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.715810    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.715928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.715892    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fn8p\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-kube-api-access-2fn8p\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.716108 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.715983    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.716108 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.716040    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.716211 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.716149    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.716283 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.716268    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.716342 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.716326    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817311 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817281    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817353    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817396    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817424    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817562    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817603    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fn8p\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-kube-api-access-2fn8p\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817649    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817701    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817722    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.817861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817802    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.818024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.817964    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.818160 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.818128    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.818288 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.818137    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.818288 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.818176    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.819802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.819783    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.820240 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.820217    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.850350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.850316    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"]
Oct 31 18:48:17.875133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.875098    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.875716 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.875655    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fn8p\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-kube-api-access-2fn8p\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:17.965664 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:17.965595    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:18.401488 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:18.401446    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"]
Oct 31 18:48:18.403161 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:48:18.403133    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9ddfe3_e3c0_4f86_b93c_b9ba98adf323.slice/crio-d7c6012d402a67221cea36ad846bcf62b958c9451d9e48201f6820ab828504db WatchSource:0}: Error finding container d7c6012d402a67221cea36ad846bcf62b958c9451d9e48201f6820ab828504db: Status 404 returned error can't find the container with id d7c6012d402a67221cea36ad846bcf62b958c9451d9e48201f6820ab828504db
Oct 31 18:48:18.405601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:18.405569    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:18.405674 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:18.405633    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:18.405674 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:18.405661    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:18.839827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:18.839754    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" event={"ID":"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323","Type":"ContainerStarted","Data":"d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a"}
Oct 31 18:48:18.839827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:18.839803    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" event={"ID":"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323","Type":"ContainerStarted","Data":"d7c6012d402a67221cea36ad846bcf62b958c9451d9e48201f6820ab828504db"}
Oct 31 18:48:18.968486 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:18.968390    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:19.075483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:19.063931    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" podStartSLOduration=2.0639107389999998 podStartE2EDuration="2.063910739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:19.005473902 +0000 UTC m=+1602.236712194" watchObservedRunningTime="2026-10-31 18:48:19.063910739 +0000 UTC m=+1602.295149031"
Oct 31 18:48:19.968485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:19.968214    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 18:48:19.968485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:19.968270    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 18:48:20.969483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:20.968863    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 18:48:20.969483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:20.968924    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 18:48:21.978007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:21.977519    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 18:48:21.978007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:21.977594    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 18:48:22.974628 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:22.974591    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 18:48:22.974832 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:22.974652    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 18:48:23.969484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:23.966527    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 18:48:23.969484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:23.966600    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 18:48:25.241039 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.241004    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"]
Oct 31 18:48:25.245142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.245119    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.427232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.424711    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"]
Oct 31 18:48:25.435641 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.435605    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-credential-socket\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.437621 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.437592    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jblqk\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-kube-api-access-jblqk\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.440187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.440162    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-socket\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.440390 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.440371    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-data\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.440651 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.440633    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-podinfo\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.440820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.440806    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-certs\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.441010 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.440995    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-envoy\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.441162 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.441146    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-token\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.441320 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.441306    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563baac-7e8c-45f1-aa23-a6eb2430341d-istiod-ca-cert\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.441676 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.436163    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5dc7b6-openshift-default-dockercfg-fgsld\""
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.544774    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-credential-socket\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.544854    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jblqk\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-kube-api-access-jblqk\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.544875    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-socket\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.544898    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-data\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.544933    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-podinfo\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.544961    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-certs\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.545008    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-envoy\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.545029    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-token\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.545057    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563baac-7e8c-45f1-aa23-a6eb2430341d-istiod-ca-cert\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.545988    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-socket\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.546248    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563baac-7e8c-45f1-aa23-a6eb2430341d-istiod-ca-cert\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.546644    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-data\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.548204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.548005    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-credential-socket\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.549591 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.549503    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-certs\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.552272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.552216    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-envoy\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.553218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.553197    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-podinfo\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.607156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.607107    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-token\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.607340 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.607231    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jblqk\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-kube-api-access-jblqk\") pod \"e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") " pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:25.614009 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.613720    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:25.615071 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.615046    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:25.616422 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.616397    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:48:25.875382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:25.875351    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:26.311538 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:26.311511    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"]
Oct 31 18:48:26.312614 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:48:26.312588    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1563baac_7e8c_45f1_aa23_a6eb2430341d.slice/crio-a624e747ca7069490d07af17e6b8a7cd90366ce9ab8f7eab8354fdb2ba607f68 WatchSource:0}: Error finding container a624e747ca7069490d07af17e6b8a7cd90366ce9ab8f7eab8354fdb2ba607f68: Status 404 returned error can't find the container with id a624e747ca7069490d07af17e6b8a7cd90366ce9ab8f7eab8354fdb2ba607f68
Oct 31 18:48:26.315321 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:26.315286    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:26.315408 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:26.315362    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:26.315408 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:26.315391    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:26.986837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:26.986795    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" event={"ID":"1563baac-7e8c-45f1-aa23-a6eb2430341d","Type":"ContainerStarted","Data":"59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf"}
Oct 31 18:48:26.987024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:26.986861    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" event={"ID":"1563baac-7e8c-45f1-aa23-a6eb2430341d","Type":"ContainerStarted","Data":"a624e747ca7069490d07af17e6b8a7cd90366ce9ab8f7eab8354fdb2ba607f68"}
Oct 31 18:48:27.104634 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:27.104574    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" podStartSLOduration=2.104555619 podStartE2EDuration="2.104555619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:27.101192163 +0000 UTC m=+1610.332430488" watchObservedRunningTime="2026-10-31 18:48:27.104555619 +0000 UTC m=+1610.335793910"
Oct 31 18:48:27.875827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:27.875785    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:27.881819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:27.881786    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:28.003535 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:28.003505    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:28.004506 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:28.004450    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:48:32.773270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.773239    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"]
Oct 31 18:48:32.777514 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.777491    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.853626 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.853560    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.853826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.853708    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.853826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.853761    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprvp\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-kube-api-access-fprvp\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.853960 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.853924    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.854088 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.853989    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.854088 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.854038    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.854202 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.854099    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.854202 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.854144    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.854310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.854223    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.955817 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.955778    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956004 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.955844    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956004 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.955926    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956106 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956031    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956106 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956067    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956114    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprvp\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-kube-api-access-fprvp\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956294    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956322    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956564 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956433    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956564 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956504    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956564 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956558    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956712 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956606    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956862 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956836    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.956918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.956887    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.959194 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.959163    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.959366 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.959341    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:32.976759 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:32.976681    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"]
Oct 31 18:48:33.012840 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:33.012802    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:33.012986 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:33.012862    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprvp\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-kube-api-access-fprvp\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:33.093967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:33.093890    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:33.521339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:33.521311    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"]
Oct 31 18:48:33.522749 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:48:33.522720    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f06fe01_12f2_4d5e_add2_b30b9aeaeb12.slice/crio-dfacc4a8de25b61e8d4bca9993876f19e783b1c4567bd7d3c19346957368566e WatchSource:0}: Error finding container dfacc4a8de25b61e8d4bca9993876f19e783b1c4567bd7d3c19346957368566e: Status 404 returned error can't find the container with id dfacc4a8de25b61e8d4bca9993876f19e783b1c4567bd7d3c19346957368566e
Oct 31 18:48:33.525537 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:33.525507    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:33.525619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:33.525571    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:33.525619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:33.525600    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:34.059705 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:34.059656    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" event={"ID":"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12","Type":"ContainerStarted","Data":"8897bb8ec178a8df99409821775ec5873dd3afcc7d5aaba7292df1f6c4aaec22"}
Oct 31 18:48:34.060212 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:34.059742    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" event={"ID":"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12","Type":"ContainerStarted","Data":"dfacc4a8de25b61e8d4bca9993876f19e783b1c4567bd7d3c19346957368566e"}
Oct 31 18:48:34.094923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:34.094886    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:34.096530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:34.096504    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 18:48:34.096673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:34.096554    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 18:48:34.200665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:34.200618    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podStartSLOduration=2.200603625 podStartE2EDuration="2.200603625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:34.198403447 +0000 UTC m=+1617.429641770" watchObservedRunningTime="2026-10-31 18:48:34.200603625 +0000 UTC m=+1617.431841972"
Oct 31 18:48:35.094445 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:35.094411    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 18:48:35.095107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:35.095002    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 18:48:36.095100 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:36.095069    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 18:48:36.095616 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:36.095122    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 18:48:37.094943 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:37.094906    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 18:48:37.095160 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:37.094965    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 18:48:38.094772 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:38.094733    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 18:48:38.094964 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:38.094797    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 18:48:39.095257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:39.095153    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 18:48:39.095257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:39.095218    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 18:48:40.182697 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:40.182661    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:40.184164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:40.184141    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:40.185032 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:40.185012    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:48:46.606067 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:48:46.605973    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:48:49.015007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.014972    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"]
Oct 31 18:48:49.020604 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.020582    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.127480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.127028    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"]
Oct 31 18:48:49.163965 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.128496    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-certs\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.235529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.235489    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istiod-ca-cert\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.245531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.182769    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2a4923-openshift-default-dockercfg-ptt25\""
Oct 31 18:48:49.245531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.239529    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-token\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.245531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.239784    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-podinfo\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.245531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.239835    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-socket\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.245531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.240102    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hkj\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-kube-api-access-26hkj\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.245531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.240133    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-envoy\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.245531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.240357    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-data\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.245531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.240534    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-credential-socket\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.342622 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342373    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-credential-socket\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.342622 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342575    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-certs\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.342872 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342635    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istiod-ca-cert\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.342872 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342668    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-token\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.342872 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342745    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-podinfo\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.342872 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342797    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-socket\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.342872 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342842    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hkj\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-kube-api-access-26hkj\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.343133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342872    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-credential-socket\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.343133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342877    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-envoy\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.343133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342912    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-certs\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.343133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.342947    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-data\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.343376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.343354    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-data\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.343586 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.343558    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-socket\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.344107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.344085    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istiod-ca-cert\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.345890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.345865    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-envoy\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.347245 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.347223    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-podinfo\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.400795 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.400756    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-token\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.400968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.400933    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hkj\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-kube-api-access-26hkj\") pod \"e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") " pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:49.634358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:49.634323    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:50.083972 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:50.083941    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"]
Oct 31 18:48:50.085010 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:48:50.084979    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e02db3a_b339_4d5c_b8f0_b1ea23f6b705.slice/crio-20f05af8caa4c83880e7e02b3f1c00a812ace7a064be76ebbb99cc07d937bdf5 WatchSource:0}: Error finding container 20f05af8caa4c83880e7e02b3f1c00a812ace7a064be76ebbb99cc07d937bdf5: Status 404 returned error can't find the container with id 20f05af8caa4c83880e7e02b3f1c00a812ace7a064be76ebbb99cc07d937bdf5
Oct 31 18:48:50.087735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:50.087704    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:50.087803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:50.087769    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:50.087803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:50.087795    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:48:50.279354 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:50.279312    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" event={"ID":"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705","Type":"ContainerStarted","Data":"6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476"}
Oct 31 18:48:50.279612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:50.279377    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" event={"ID":"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705","Type":"ContainerStarted","Data":"20f05af8caa4c83880e7e02b3f1c00a812ace7a064be76ebbb99cc07d937bdf5"}
Oct 31 18:48:50.396961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:50.396915    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" podStartSLOduration=2.396901479 podStartE2EDuration="2.396901479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:50.394612446 +0000 UTC m=+1633.625850737" watchObservedRunningTime="2026-10-31 18:48:50.396901479 +0000 UTC m=+1633.628139768"
Oct 31 18:48:50.634890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:50.634848    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:51.640475 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:51.640422    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:52.292224 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:52.292187    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:48:52.293107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:48:52.293088    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:49:10.301335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:10.301300    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"]
Oct 31 18:49:10.301858 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:10.301548    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy" containerID="cri-o://112aeb7b4cfb85fc3365226f1b291c4b2a3a10e6fe7e68bdb214f17e97ffcd5a" gracePeriod=30
Oct 31 18:49:10.351531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:10.351492    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"]
Oct 31 18:49:10.351982 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:10.351782    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" containerID="cri-o://ec813721f13529db9af15bcc7eaa139ee724879a3e98619b5b745526f9e28ecb" gracePeriod=30
Oct 31 18:49:10.355306 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:10.355269    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"]
Oct 31 18:49:10.355510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:10.355485    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" containerID="cri-o://c4773ab8d32d17d28a7039cd7f73d21753c64c44e74c30ca7022c895ed75f5ea" gracePeriod=30
Oct 31 18:49:10.399287 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:10.399241    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"]
Oct 31 18:49:10.399529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:10.399492    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" containerID="cri-o://786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302" gracePeriod=30
Oct 31 18:49:11.867544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:11.867506    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tk5s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:49:11.868059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:11.867577    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:49:13.477192 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:13.477151    5551 generic.go:356] "Generic (PLEG): container finished" podID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerID="1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2" exitCode=0
Oct 31 18:49:13.477764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:13.477229    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" event={"ID":"6038f7c4-9cd7-445a-bde8-952014f798fb","Type":"ContainerDied","Data":"1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2"}
Oct 31 18:49:13.477832 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:13.477808    5551 scope.go:122] "RemoveContainer" containerID="1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2"
Oct 31 18:49:13.480116 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:13.480084    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:49:13.480279 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:13.480167    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:49:13.480279 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:13.480203    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:49:14.487368 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:14.487334    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" event={"ID":"6038f7c4-9cd7-445a-bde8-952014f798fb","Type":"ContainerStarted","Data":"79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158"}
Oct 31 18:49:14.918075 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:14.918033    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dsbtv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:49:14.918246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:14.918102    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:49:15.045800 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.045765    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rzjwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:49:15.045969 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.045830    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:49:15.120136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.119420    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:49:15.120136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.119579    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 18:49:15.120136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.119625    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 18:49:15.180718 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.180626    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zns87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:49:15.180886 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.180712    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:49:15.499837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.499805    5551 generic.go:356] "Generic (PLEG): container finished" podID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerID="ec813721f13529db9af15bcc7eaa139ee724879a3e98619b5b745526f9e28ecb" exitCode=0
Oct 31 18:49:15.500385 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.499850    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" event={"ID":"c8ac852a-0582-422c-bc43-6f0820bcdbb1","Type":"ContainerDied","Data":"ec813721f13529db9af15bcc7eaa139ee724879a3e98619b5b745526f9e28ecb"}
Oct 31 18:49:15.502577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.502549    5551 generic.go:356] "Generic (PLEG): container finished" podID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerID="c4773ab8d32d17d28a7039cd7f73d21753c64c44e74c30ca7022c895ed75f5ea" exitCode=0
Oct 31 18:49:15.502692 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.502663    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" event={"ID":"2e415b36-4318-42d4-a36c-ca99f48cfd7f","Type":"ContainerDied","Data":"c4773ab8d32d17d28a7039cd7f73d21753c64c44e74c30ca7022c895ed75f5ea"}
Oct 31 18:49:15.504431 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.504397    5551 generic.go:356] "Generic (PLEG): container finished" podID="dff2b758-259f-476d-85f6-1785a52e8b67" containerID="112aeb7b4cfb85fc3365226f1b291c4b2a3a10e6fe7e68bdb214f17e97ffcd5a" exitCode=0
Oct 31 18:49:15.505153 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.505119    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" event={"ID":"dff2b758-259f-476d-85f6-1785a52e8b67","Type":"ContainerDied","Data":"112aeb7b4cfb85fc3365226f1b291c4b2a3a10e6fe7e68bdb214f17e97ffcd5a"}
Oct 31 18:49:15.746046 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.745798    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:49:15.797007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.796976    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.814738    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-data\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.814809    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-istio-token\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-istio-token\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.814869    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-credential-socket\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.814962    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-envoy\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.815006    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-certs\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.815082    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-kube-api-access-fpdnb\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-kube-api-access-fpdnb\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.815174    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dff2b758-259f-476d-85f6-1785a52e8b67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff2b758-259f-476d-85f6-1785a52e8b67-istio-podinfo\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.815265    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-socket\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.815509 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.815337    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff2b758-259f-476d-85f6-1785a52e8b67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff2b758-259f-476d-85f6-1785a52e8b67-istiod-ca-cert\") pod \"dff2b758-259f-476d-85f6-1785a52e8b67\" (UID: \"dff2b758-259f-476d-85f6-1785a52e8b67\") "
Oct 31 18:49:15.816251 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.816222    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff2b758-259f-476d-85f6-1785a52e8b67-istiod-ca-cert" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:15.816890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.816650    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-certs" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.816890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.816848    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-socket" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.817793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.817759    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-credential-socket" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.818104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.818070    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-data" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.818808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.818783    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-envoy" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.820152 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.820129    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-kube-api-access-fpdnb" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "kube-api-access-fpdnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:15.822429 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.822378    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-istio-token" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:15.822970 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.822928    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dff2b758-259f-476d-85f6-1785a52e8b67-istio-podinfo" pod "dff2b758-259f-476d-85f6-1785a52e8b67" (UID: "dff2b758-259f-476d-85f6-1785a52e8b67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:49:15.902772 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.902606    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:49:15.906153 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.906121    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:49:15.916505 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.916360    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-token\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-token\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.916446    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-certs\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.916803    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-credential-socket\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.916817    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-certs" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.916861    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-data\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.916935    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-podinfo\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.916983    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-envoy\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.916994    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-credential-socket" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917027    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istiod-ca-cert\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917081    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-data" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917134    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-socket\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917172    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-kube-api-access-xk7wq\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-kube-api-access-xk7wq\") pod \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\" (UID: \"c8ac852a-0582-422c-bc43-6f0820bcdbb1\") "
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917411    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-socket" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917429    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istiod-ca-cert" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917707    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917730    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917749    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917763    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917776    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpdnb\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-kube-api-access-fpdnb\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917790    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917803    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff2b758-259f-476d-85f6-1785a52e8b67-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.917813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917815    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.918104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917829    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.918104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917841    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff2b758-259f-476d-85f6-1785a52e8b67-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.918104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917853    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.918104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917866    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff2b758-259f-476d-85f6-1785a52e8b67-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.918104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917880    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff2b758-259f-476d-85f6-1785a52e8b67-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.918104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.917894    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:15.919417 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.919390    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-envoy" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:15.919417 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.919396    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-token" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:15.919762 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.919744    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-kube-api-access-xk7wq" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "kube-api-access-xk7wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:15.920045 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:15.920023    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-podinfo" pod "c8ac852a-0582-422c-bc43-6f0820bcdbb1" (UID: "c8ac852a-0582-422c-bc43-6f0820bcdbb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.026341    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-token\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-token\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.026841    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-data\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.026881    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-certs\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.026921    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istiod-ca-cert\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.026955    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-credential-socket\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027029    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istiod-ca-cert\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027071    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-socket\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027101    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-kube-api-access-kkrnm\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-kube-api-access-kkrnm\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027174    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-envoy\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027237    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-token\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-token\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027268    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-socket\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027358    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-envoy\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027389    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-podinfo\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027426    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-data\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027488    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-credential-socket\") pod \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\" (UID: \"2e415b36-4318-42d4-a36c-ca99f48cfd7f\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027526    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-podinfo\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027575    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-kube-api-access-26shq\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-kube-api-access-26shq\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.027633    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-certs\") pod \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\" (UID: \"de6edefc-ce6a-4e9f-80e7-e39bb4e68550\") "
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.028411    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.028435    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk7wq\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-kube-api-access-xk7wq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.028451    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.028869    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ac852a-0582-422c-bc43-6f0820bcdbb1-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.029916    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-credential-socket" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.030722    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-data" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.030748    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istiod-ca-cert" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:16.031183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.030896    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-certs" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.039184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.032711    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-socket" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.039184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.033525    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-socket" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.039184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.033959    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-token" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:16.039184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.037651    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-credential-socket" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.039184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.038710    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istiod-ca-cert" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:16.039184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.039106    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-data" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.040204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.040160    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-certs" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.040895 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.040868    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-kube-api-access-kkrnm" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "kube-api-access-kkrnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:16.044131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.044097    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-envoy" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.044556 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.044537    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-envoy" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:16.044759 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.044734    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-podinfo" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:49:16.044905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.044882    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-podinfo" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:49:16.045058 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.045034    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-token" pod "2e415b36-4318-42d4-a36c-ca99f48cfd7f" (UID: "2e415b36-4318-42d4-a36c-ca99f48cfd7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:16.046876 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.046847    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-kube-api-access-26shq" pod "de6edefc-ce6a-4e9f-80e7-e39bb4e68550" (UID: "de6edefc-ce6a-4e9f-80e7-e39bb4e68550"). InnerVolumeSpecName "kube-api-access-26shq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:16.108114 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.108078    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 18:49:16.108305 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.108136    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 18:49:16.131172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131133    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131173    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131187    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131202    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131216    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131229    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131280    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131293    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131306    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26shq\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-kube-api-access-26shq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131320    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131334    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131348    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131361    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e415b36-4318-42d4-a36c-ca99f48cfd7f-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131375    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e415b36-4318-42d4-a36c-ca99f48cfd7f-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131388    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131401    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131413    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6edefc-ce6a-4e9f-80e7-e39bb4e68550-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.131438 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.131437    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrnm\" (UniqueName: \"kubernetes.io/projected/2e415b36-4318-42d4-a36c-ca99f48cfd7f-kube-api-access-kkrnm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:16.513668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.513634    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"
Oct 31 18:49:16.513668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.513658    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87" event={"ID":"2e415b36-4318-42d4-a36c-ca99f48cfd7f","Type":"ContainerDied","Data":"ec2be177738eb6ed83f32d6367910cdfcc872b71d9db08bb23c5ddd290477e6e"}
Oct 31 18:49:16.514201 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.513728    5551 scope.go:122] "RemoveContainer" containerID="c4773ab8d32d17d28a7039cd7f73d21753c64c44e74c30ca7022c895ed75f5ea"
Oct 31 18:49:16.515612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.515591    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"
Oct 31 18:49:16.515729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.515617    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8" event={"ID":"dff2b758-259f-476d-85f6-1785a52e8b67","Type":"ContainerDied","Data":"6b9202119e746bfe6f25359f49c2b4d3afc5152e83b80d15252d57a6ea406b31"}
Oct 31 18:49:16.524192 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.524162    5551 generic.go:356] "Generic (PLEG): container finished" podID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerID="786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302" exitCode=0
Oct 31 18:49:16.524320 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.524214    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"
Oct 31 18:49:16.524320 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.524237    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" event={"ID":"de6edefc-ce6a-4e9f-80e7-e39bb4e68550","Type":"ContainerDied","Data":"786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302"}
Oct 31 18:49:16.524320 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.524299    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv" event={"ID":"de6edefc-ce6a-4e9f-80e7-e39bb4e68550","Type":"ContainerDied","Data":"ce48068027cd9f51b71f16f0866be99d4d53782b0458aa6239b571d8aedc28e7"}
Oct 31 18:49:16.526236 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.526135    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"
Oct 31 18:49:16.526236 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.526180    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf" event={"ID":"c8ac852a-0582-422c-bc43-6f0820bcdbb1","Type":"ContainerDied","Data":"b97ee80229b57824349381b14014e4c4dc8562a56c08ac00f42ec068a373a44f"}
Oct 31 18:49:16.527685 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.527631    5551 scope.go:122] "RemoveContainer" containerID="112aeb7b4cfb85fc3365226f1b291c4b2a3a10e6fe7e68bdb214f17e97ffcd5a"
Oct 31 18:49:16.538289 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.538254    5551 scope.go:122] "RemoveContainer" containerID="786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302"
Oct 31 18:49:16.548961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.548941    5551 scope.go:122] "RemoveContainer" containerID="786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302"
Oct 31 18:49:16.549313 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:49:16.549291    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302\": container with ID starting with 786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302 not found: ID does not exist" containerID="786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302"
Oct 31 18:49:16.549373 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.549325    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302"} err="failed to get container status \"786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302\": rpc error: code = NotFound desc = could not find container \"786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302\": container with ID starting with 786ca720868b4bb6b24b3dc345d85ec9cd4a7e4f7ac98301f254a409bc394302 not found: ID does not exist"
Oct 31 18:49:16.549373 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.549344    5551 scope.go:122] "RemoveContainer" containerID="ec813721f13529db9af15bcc7eaa139ee724879a3e98619b5b745526f9e28ecb"
Oct 31 18:49:16.608351 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:49:16.608306    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:49:16.685827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.685793    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"]
Oct 31 18:49:16.735028 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.734997    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zns87"]
Oct 31 18:49:16.844003 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.843929    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"]
Oct 31 18:49:16.848805 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.848779    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dsbtv"]
Oct 31 18:49:16.910301 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.910269    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" path="/var/lib/kubelet/pods/2e415b36-4318-42d4-a36c-ca99f48cfd7f/volumes"
Oct 31 18:49:16.910807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.910790    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" path="/var/lib/kubelet/pods/de6edefc-ce6a-4e9f-80e7-e39bb4e68550/volumes"
Oct 31 18:49:16.994713 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:16.994669    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"]
Oct 31 18:49:17.001773 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:17.001746    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rzjwf"]
Oct 31 18:49:17.085820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:17.085787    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"]
Oct 31 18:49:17.088722 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:17.088698    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tk5s8"]
Oct 31 18:49:17.108323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:17.108254    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 18:49:17.108433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:17.108309    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 18:49:18.108189 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:18.108154    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 18:49:18.108629 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:18.108214    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 18:49:18.946606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:18.946568    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" path="/var/lib/kubelet/pods/c8ac852a-0582-422c-bc43-6f0820bcdbb1/volumes"
Oct 31 18:49:18.947098 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:18.947078    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" path="/var/lib/kubelet/pods/dff2b758-259f-476d-85f6-1785a52e8b67/volumes"
Oct 31 18:49:19.109482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.108832    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 18:49:19.109482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.108905    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 18:49:19.881298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881262    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"]
Oct 31 18:49:19.881862 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881847    5551 cpu_manager.go:414] "containerMap: removing container" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy"
Oct 31 18:49:19.881915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881864    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy"
Oct 31 18:49:19.881915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881879    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy"
Oct 31 18:49:19.881915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881884    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy"
Oct 31 18:49:19.881915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881894    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy"
Oct 31 18:49:19.881915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881899    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy"
Oct 31 18:49:19.882059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881925    5551 cpu_manager.go:414] "containerMap: removing container" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy"
Oct 31 18:49:19.882059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.881929    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy"
Oct 31 18:49:19.882059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.882015    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6edefc-ce6a-4e9f-80e7-e39bb4e68550" containerName="istio-proxy"
Oct 31 18:49:19.882059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.882027    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff2b758-259f-476d-85f6-1785a52e8b67" containerName="istio-proxy"
Oct 31 18:49:19.882059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.882034    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ac852a-0582-422c-bc43-6f0820bcdbb1" containerName="istio-proxy"
Oct 31 18:49:19.882059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.882041    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e415b36-4318-42d4-a36c-ca99f48cfd7f" containerName="istio-proxy"
Oct 31 18:49:19.887119 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.887099    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:19.942689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.942661    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"]
Oct 31 18:49:19.996513 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:19.988259    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5dc7b6-serving-cert\""
Oct 31 18:49:20.019647 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.019543    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.020420 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.020389    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5q9\" (UniqueName: \"kubernetes.io/projected/4d23327e-95c2-432f-b8ba-cdb835d739ee-kube-api-access-rs5q9\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.022616 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.020604    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.022616 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.021441    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d23327e-95c2-432f-b8ba-cdb835d739ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.112579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.112554    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:49:20.112992 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.112977    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:49:20.113683 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.113666    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:49:20.122237 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.122212    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.122356 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.122259    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5q9\" (UniqueName: \"kubernetes.io/projected/4d23327e-95c2-432f-b8ba-cdb835d739ee-kube-api-access-rs5q9\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.122356 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.122336    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.122495 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.122419    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d23327e-95c2-432f-b8ba-cdb835d739ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.123114 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.123096    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d23327e-95c2-432f-b8ba-cdb835d739ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.124849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.124828    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.125033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.125013    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.179052 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.178983    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5q9\" (UniqueName: \"kubernetes.io/projected/4d23327e-95c2-432f-b8ba-cdb835d739ee-kube-api-access-rs5q9\") pod \"maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.209482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.204180    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:20.609753 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:20.609730    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"]
Oct 31 18:49:20.612079 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:49:20.612049    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d23327e_95c2_432f_b8ba_cdb835d739ee.slice/crio-9c9100d46b04b55bb722ef5b8e075ec064c750289c40b0b358579bcb3ad8b660 WatchSource:0}: Error finding container 9c9100d46b04b55bb722ef5b8e075ec064c750289c40b0b358579bcb3ad8b660: Status 404 returned error can't find the container with id 9c9100d46b04b55bb722ef5b8e075ec064c750289c40b0b358579bcb3ad8b660
Oct 31 18:49:21.611041 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:21.611005    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd" event={"ID":"4d23327e-95c2-432f-b8ba-cdb835d739ee","Type":"ContainerStarted","Data":"3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c"}
Oct 31 18:49:21.611546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:21.611255    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:21.611546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:21.611299    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd" event={"ID":"4d23327e-95c2-432f-b8ba-cdb835d739ee","Type":"ContainerStarted","Data":"9c9100d46b04b55bb722ef5b8e075ec064c750289c40b0b358579bcb3ad8b660"}
Oct 31 18:49:21.725644 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:21.725546    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd" podStartSLOduration=1.915152547 podStartE2EDuration="2.72553367s" totalImagesPullingTime="810.381123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:49:21.724930571 +0000 UTC m=+1664.956168864" watchObservedRunningTime="2026-10-31 18:49:21.72553367 +0000 UTC m=+1664.956771960"
Oct 31 18:49:27.624280 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:27.624252    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:49:34.159699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.159669    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"]
Oct 31 18:49:34.160191 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.159853    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q" podUID="018b2948-32ed-4f73-8be5-e4a390a6079f" containerName="maas-api" containerID="cri-o://7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63" gracePeriod=30
Oct 31 18:49:34.465696 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.465670    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:49:34.555485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.554854    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/018b2948-32ed-4f73-8be5-e4a390a6079f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/018b2948-32ed-4f73-8be5-e4a390a6079f-openshift-service-ca\") pod \"018b2948-32ed-4f73-8be5-e4a390a6079f\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") "
Oct 31 18:49:34.555485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.554914    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-metrics-certs\") pod \"018b2948-32ed-4f73-8be5-e4a390a6079f\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") "
Oct 31 18:49:34.555485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.554983    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls\") pod \"018b2948-32ed-4f73-8be5-e4a390a6079f\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") "
Oct 31 18:49:34.562481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.556532    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018b2948-32ed-4f73-8be5-e4a390a6079f-openshift-service-ca" pod "018b2948-32ed-4f73-8be5-e4a390a6079f" (UID: "018b2948-32ed-4f73-8be5-e4a390a6079f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:34.562481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.556760    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018b2948-32ed-4f73-8be5-e4a390a6079f-kube-api-access-6s5cg\" (UniqueName: \"kubernetes.io/projected/018b2948-32ed-4f73-8be5-e4a390a6079f-kube-api-access-6s5cg\") pod \"018b2948-32ed-4f73-8be5-e4a390a6079f\" (UID: \"018b2948-32ed-4f73-8be5-e4a390a6079f\") "
Oct 31 18:49:34.562481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.557272    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/018b2948-32ed-4f73-8be5-e4a390a6079f-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:34.577046 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.575684    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls" pod "018b2948-32ed-4f73-8be5-e4a390a6079f" (UID: "018b2948-32ed-4f73-8be5-e4a390a6079f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:49:34.585529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.585367    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018b2948-32ed-4f73-8be5-e4a390a6079f-kube-api-access-6s5cg" pod "018b2948-32ed-4f73-8be5-e4a390a6079f" (UID: "018b2948-32ed-4f73-8be5-e4a390a6079f"). InnerVolumeSpecName "kube-api-access-6s5cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:34.592875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.592817    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-metrics-certs" pod "018b2948-32ed-4f73-8be5-e4a390a6079f" (UID: "018b2948-32ed-4f73-8be5-e4a390a6079f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:49:34.658672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.658637    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s5cg\" (UniqueName: \"kubernetes.io/projected/018b2948-32ed-4f73-8be5-e4a390a6079f-kube-api-access-6s5cg\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:34.658672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.658670    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:34.658876 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.658683    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/018b2948-32ed-4f73-8be5-e4a390a6079f-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:34.720869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.720757    5551 generic.go:356] "Generic (PLEG): container finished" podID="018b2948-32ed-4f73-8be5-e4a390a6079f" containerID="7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63" exitCode=0
Oct 31 18:49:34.721043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.720895    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q" event={"ID":"018b2948-32ed-4f73-8be5-e4a390a6079f","Type":"ContainerDied","Data":"7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63"}
Oct 31 18:49:34.721043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.720948    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q" event={"ID":"018b2948-32ed-4f73-8be5-e4a390a6079f","Type":"ContainerDied","Data":"32dddc8338b21ba33ef4e73077fda2116e92dec875f9ef68d1034b107a3672c3"}
Oct 31 18:49:34.721043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.720976    5551 scope.go:122] "RemoveContainer" containerID="7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63"
Oct 31 18:49:34.722648 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.721141    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"
Oct 31 18:49:34.734966 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.734358    5551 scope.go:122] "RemoveContainer" containerID="7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63"
Oct 31 18:49:34.735870 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:49:34.735570    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63\": container with ID starting with 7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63 not found: ID does not exist" containerID="7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63"
Oct 31 18:49:34.735870 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.735611    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63"} err="failed to get container status \"7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63\": rpc error: code = NotFound desc = could not find container \"7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63\": container with ID starting with 7f49f2dc6a3e4272bff26754619cb7b956553fa11ecb0ad4119e7d54ed13ad63 not found: ID does not exist"
Oct 31 18:49:34.886394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.886358    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"]
Oct 31 18:49:34.889765 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.889732    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c21e9e-64785468bf-w4p8q"]
Oct 31 18:49:34.909561 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:34.909526    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="018b2948-32ed-4f73-8be5-e4a390a6079f" path="/var/lib/kubelet/pods/018b2948-32ed-4f73-8be5-e4a390a6079f/volumes"
Oct 31 18:49:42.610415 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:42.610377    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"]
Oct 31 18:49:42.610948 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:42.610704    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy" containerID="cri-o://6a6de976ff5eec30f6f0b4febc087d537e45e7e038e54eb4f432d81e26d431d8" gracePeriod=30
Oct 31 18:49:45.403394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.403356    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"]
Oct 31 18:49:45.403940 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.403925    5551 cpu_manager.go:414] "containerMap: removing container" podUID="018b2948-32ed-4f73-8be5-e4a390a6079f" containerName="maas-api"
Oct 31 18:49:45.403987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.403941    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018b2948-32ed-4f73-8be5-e4a390a6079f" containerName="maas-api"
Oct 31 18:49:45.404065 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.404056    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="018b2948-32ed-4f73-8be5-e4a390a6079f" containerName="maas-api"
Oct 31 18:49:45.409130 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.409113    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.464524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.464497    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"]
Oct 31 18:49:45.494864 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.494829    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.495024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.494948    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppg9p\" (UniqueName: \"kubernetes.io/projected/d87eb73b-5383-423c-9496-54c6a1741afa-kube-api-access-ppg9p\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.495024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.494977    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87eb73b-5383-423c-9496-54c6a1741afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.495152 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.495138    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-metrics-certs\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.509813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.509791    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2a4923-serving-cert\""
Oct 31 18:49:45.596313 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.596281    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.596510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.596395    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppg9p\" (UniqueName: \"kubernetes.io/projected/d87eb73b-5383-423c-9496-54c6a1741afa-kube-api-access-ppg9p\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.596510 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:49:45.596436    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-serving-cert: secret "maas-api-e2e-ipp-a-2a4923-serving-cert" not found
Oct 31 18:49:45.596649 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:49:45.596515    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls podName:d87eb73b-5383-423c-9496-54c6a1741afa nodeName:}" failed. No retries permitted until 2026-10-31 18:49:46.096499985 +0000 UTC m=+1689.327738258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls") pod "maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh" (UID: "d87eb73b-5383-423c-9496-54c6a1741afa") : secret "maas-api-e2e-ipp-a-2a4923-serving-cert" not found
Oct 31 18:49:45.596649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.596438    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87eb73b-5383-423c-9496-54c6a1741afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.596786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.596697    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-metrics-certs\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.597114 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.597091    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87eb73b-5383-423c-9496-54c6a1741afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.599086 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.599066    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-metrics-certs\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:45.656867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:45.656747    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppg9p\" (UniqueName: \"kubernetes.io/projected/d87eb73b-5383-423c-9496-54c6a1741afa-kube-api-access-ppg9p\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:46.111372 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:46.111278    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:46.114329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:46.114302    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:46.323328 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:46.323293    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:46.605174 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:49:46.605135    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:49:46.706571 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:46.706539    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"]
Oct 31 18:49:46.707573 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:49:46.707542    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87eb73b_5383_423c_9496_54c6a1741afa.slice/crio-1fbf670f4f9fb84864be8ddd83cdb365de1302b66caa4c75f68352dd316cf3d6 WatchSource:0}: Error finding container 1fbf670f4f9fb84864be8ddd83cdb365de1302b66caa4c75f68352dd316cf3d6: Status 404 returned error can't find the container with id 1fbf670f4f9fb84864be8ddd83cdb365de1302b66caa4c75f68352dd316cf3d6
Oct 31 18:49:46.826819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:46.826791    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh" event={"ID":"d87eb73b-5383-423c-9496-54c6a1741afa","Type":"ContainerStarted","Data":"1fbf670f4f9fb84864be8ddd83cdb365de1302b66caa4c75f68352dd316cf3d6"}
Oct 31 18:49:47.838290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:47.838255    5551 generic.go:356] "Generic (PLEG): container finished" podID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerID="6a6de976ff5eec30f6f0b4febc087d537e45e7e038e54eb4f432d81e26d431d8" exitCode=0
Oct 31 18:49:47.838290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:47.838283    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" event={"ID":"95a768f9-2b7c-43f0-8f36-f59966d129ae","Type":"ContainerDied","Data":"6a6de976ff5eec30f6f0b4febc087d537e45e7e038e54eb4f432d81e26d431d8"}
Oct 31 18:49:47.838852 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:47.838342    5551 scope.go:122] "RemoveContainer" containerID="45f8d89440a9c0841b8e391e1ad5c6d6ab442d449431864c88f6d1b57ba57dce"
Oct 31 18:49:47.843649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:47.843592    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh" event={"ID":"d87eb73b-5383-423c-9496-54c6a1741afa","Type":"ContainerStarted","Data":"47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819"}
Oct 31 18:49:47.844414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:47.844365    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:49:47.888799 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:47.888775    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:49:47.958222 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:47.958169    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh" podStartSLOduration=2.150294148 podStartE2EDuration="2.958154466s" totalImagesPullingTime="807.860318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:49:47.957347418 +0000 UTC m=+1691.188585709" watchObservedRunningTime="2026-10-31 18:49:47.958154466 +0000 UTC m=+1691.189392756"
Oct 31 18:49:48.034957 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.034862    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-token\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-token\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.034974    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-data\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035002    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-envoy\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035028    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-certs\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035116    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-socket\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035162    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-credential-socket\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035223    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a768f9-2b7c-43f0-8f36-f59966d129ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a768f9-2b7c-43f0-8f36-f59966d129ae-istiod-ca-cert\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035339    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-kube-api-access-8n785\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-kube-api-access-8n785\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035354    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-data" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:48.035542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035403    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-podinfo\") pod \"95a768f9-2b7c-43f0-8f36-f59966d129ae\" (UID: \"95a768f9-2b7c-43f0-8f36-f59966d129ae\") "
Oct 31 18:49:48.035654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035593    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-socket" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:48.035804 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035778    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a768f9-2b7c-43f0-8f36-f59966d129ae-istiod-ca-cert" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:48.036043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035974    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-certs" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:48.036043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.035978    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-credential-socket" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:48.036217 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.036190    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.036294 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.036222    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.036294 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.036238    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.036294 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.036254    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.036294 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.036270    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a768f9-2b7c-43f0-8f36-f59966d129ae-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.037704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.037679    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-envoy" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:49:48.038259 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.038213    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-kube-api-access-8n785" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "kube-api-access-8n785". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:48.038365 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.038310    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-podinfo" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:49:48.038590 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.038566    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-token" pod "95a768f9-2b7c-43f0-8f36-f59966d129ae" (UID: "95a768f9-2b7c-43f0-8f36-f59966d129ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:48.137913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.137882    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.137913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.137908    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.137913 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.137918    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n785\" (UniqueName: \"kubernetes.io/projected/95a768f9-2b7c-43f0-8f36-f59966d129ae-kube-api-access-8n785\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.138282 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.137928    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a768f9-2b7c-43f0-8f36-f59966d129ae-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:48.853628 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.853580    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc" event={"ID":"95a768f9-2b7c-43f0-8f36-f59966d129ae","Type":"ContainerDied","Data":"ef140d4f2eaae60d3b0a4af0975647b897bf6189c88c409fa649e9f00c5afcd2"}
Oct 31 18:49:48.854124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.853651    5551 scope.go:122] "RemoveContainer" containerID="6a6de976ff5eec30f6f0b4febc087d537e45e7e038e54eb4f432d81e26d431d8"
Oct 31 18:49:48.854199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:48.854138    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"
Oct 31 18:49:49.033806 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:49.033774    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"]
Oct 31 18:49:49.096493 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:49.096419    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c21e9e-openshift-default-b94bc6dcf-5rfmc"]
Oct 31 18:49:50.909003 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:50.908971    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" path="/var/lib/kubelet/pods/95a768f9-2b7c-43f0-8f36-f59966d129ae/volumes"
Oct 31 18:49:53.859691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:49:53.859658    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:50:00.119706 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.119669    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"]
Oct 31 18:50:00.120134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.119886    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p" podUID="8ac7383d-186a-48eb-923e-0faea300f803" containerName="maas-api" containerID="cri-o://d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc" gracePeriod=30
Oct 31 18:50:00.507389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.507360    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:50:00.540290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.540251    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac7383d-186a-48eb-923e-0faea300f803-kube-api-access-r2tq7\" (UniqueName: \"kubernetes.io/projected/8ac7383d-186a-48eb-923e-0faea300f803-kube-api-access-r2tq7\") pod \"8ac7383d-186a-48eb-923e-0faea300f803\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") "
Oct 31 18:50:00.540433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.540367    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls\") pod \"8ac7383d-186a-48eb-923e-0faea300f803\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") "
Oct 31 18:50:00.540433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.540390    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-metrics-certs\") pod \"8ac7383d-186a-48eb-923e-0faea300f803\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") "
Oct 31 18:50:00.540531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.540442    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac7383d-186a-48eb-923e-0faea300f803-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac7383d-186a-48eb-923e-0faea300f803-openshift-service-ca\") pod \"8ac7383d-186a-48eb-923e-0faea300f803\" (UID: \"8ac7383d-186a-48eb-923e-0faea300f803\") "
Oct 31 18:50:00.541328 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.541294    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac7383d-186a-48eb-923e-0faea300f803-openshift-service-ca" pod "8ac7383d-186a-48eb-923e-0faea300f803" (UID: "8ac7383d-186a-48eb-923e-0faea300f803"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:00.542704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.542660    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac7383d-186a-48eb-923e-0faea300f803-kube-api-access-r2tq7" pod "8ac7383d-186a-48eb-923e-0faea300f803" (UID: "8ac7383d-186a-48eb-923e-0faea300f803"). InnerVolumeSpecName "kube-api-access-r2tq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:00.542793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.542698    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls" pod "8ac7383d-186a-48eb-923e-0faea300f803" (UID: "8ac7383d-186a-48eb-923e-0faea300f803"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:50:00.542793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.542742    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-metrics-certs" pod "8ac7383d-186a-48eb-923e-0faea300f803" (UID: "8ac7383d-186a-48eb-923e-0faea300f803"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:50:00.646091 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.646061    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2tq7\" (UniqueName: \"kubernetes.io/projected/8ac7383d-186a-48eb-923e-0faea300f803-kube-api-access-r2tq7\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:00.646270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.646258    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:00.646332 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.646325    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac7383d-186a-48eb-923e-0faea300f803-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:00.646387 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.646380    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac7383d-186a-48eb-923e-0faea300f803-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:00.947916 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.947886    5551 generic.go:356] "Generic (PLEG): container finished" podID="8ac7383d-186a-48eb-923e-0faea300f803" containerID="d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc" exitCode=0
Oct 31 18:50:00.948139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.947940    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"
Oct 31 18:50:00.948139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.947966    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p" event={"ID":"8ac7383d-186a-48eb-923e-0faea300f803","Type":"ContainerDied","Data":"d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc"}
Oct 31 18:50:00.948139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.948022    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p" event={"ID":"8ac7383d-186a-48eb-923e-0faea300f803","Type":"ContainerDied","Data":"3712b5f5c7e015f93a1889b829cf310172f05e4d87598918b70546a2976d4185"}
Oct 31 18:50:00.948139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.948043    5551 scope.go:122] "RemoveContainer" containerID="d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc"
Oct 31 18:50:00.992001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.991604    5551 scope.go:122] "RemoveContainer" containerID="d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc"
Oct 31 18:50:00.992540 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:50:00.992451    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc\": container with ID starting with d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc not found: ID does not exist" containerID="d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc"
Oct 31 18:50:00.992540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:00.992508    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc"} err="failed to get container status \"d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc\": rpc error: code = NotFound desc = could not find container \"d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc\": container with ID starting with d53b862c304fd3a80551c7795d5e94bd95338144ebbe2d28aeb4c667a12f15fc not found: ID does not exist"
Oct 31 18:50:01.131684 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:01.131657    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"]
Oct 31 18:50:01.166184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:01.166147    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3417b822-764cd85fdd-xt29p"]
Oct 31 18:50:02.908743 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:02.908712    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac7383d-186a-48eb-923e-0faea300f803" path="/var/lib/kubelet/pods/8ac7383d-186a-48eb-923e-0faea300f803/volumes"
Oct 31 18:50:04.047842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:04.047807    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"]
Oct 31 18:50:04.099090 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:04.099057    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fad6d17-nmjml"]
Oct 31 18:50:04.909928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:04.909886    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0e48de-3861-4641-87d8-91d32549def1" path="/var/lib/kubelet/pods/0c0e48de-3861-4641-87d8-91d32549def1/volumes"
Oct 31 18:50:05.991165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:05.991134    5551 generic.go:356] "Generic (PLEG): container finished" podID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerID="59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf" exitCode=0
Oct 31 18:50:05.991620 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:05.991208    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" event={"ID":"1563baac-7e8c-45f1-aa23-a6eb2430341d","Type":"ContainerDied","Data":"59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf"}
Oct 31 18:50:05.991620 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:05.991616    5551 scope.go:122] "RemoveContainer" containerID="59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf"
Oct 31 18:50:05.993133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:05.993106    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:05.993195 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:05.993171    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:05.993232 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:05.993203    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:06.694003 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:06.693968    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"]
Oct 31 18:50:06.694209 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:06.694186    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy" containerID="cri-o://d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a" gracePeriod=30
Oct 31 18:50:06.710346 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:06.710309    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"]
Oct 31 18:50:06.710615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:06.710588    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy" containerID="cri-o://8897bb8ec178a8df99409821775ec5873dd3afcc7d5aaba7292df1f6c4aaec22" gracePeriod=30
Oct 31 18:50:06.739312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:06.739275    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"]
Oct 31 18:50:06.739588 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:06.739558    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy" containerID="cri-o://e7fb9169eb0dd25cb5e2fca2fd2497747e742175dd9694090fabe6cd5fbb05fa" gracePeriod=30
Oct 31 18:50:07.000851 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.000759    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" event={"ID":"1563baac-7e8c-45f1-aa23-a6eb2430341d","Type":"ContainerStarted","Data":"1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3"}
Oct 31 18:50:07.280825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.280739    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"]
Oct 31 18:50:07.281405 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.281383    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8ac7383d-186a-48eb-923e-0faea300f803" containerName="maas-api"
Oct 31 18:50:07.281573 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.281405    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac7383d-186a-48eb-923e-0faea300f803" containerName="maas-api"
Oct 31 18:50:07.281573 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.281435    5551 cpu_manager.go:414] "containerMap: removing container" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy"
Oct 31 18:50:07.281573 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.281442    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy"
Oct 31 18:50:07.281803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.281646    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy"
Oct 31 18:50:07.281803 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.281678    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac7383d-186a-48eb-923e-0faea300f803" containerName="maas-api"
Oct 31 18:50:07.285042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.285020    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:07.338150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.338100    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5d1b3f\""
Oct 31 18:50:07.338363 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.338338    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5d1b3f-dockercfg-pj6ss\""
Oct 31 18:50:07.340949 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.340927    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"]
Oct 31 18:50:07.395359 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.395316    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtfbp\" (UniqueName: \"kubernetes.io/projected/66279707-0137-4a71-87fb-da0938885850-kube-api-access-rtfbp\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr\" (UID: \"66279707-0137-4a71-87fb-da0938885850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:07.395619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.395595    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66279707-0137-4a71-87fb-da0938885850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr\" (UID: \"66279707-0137-4a71-87fb-da0938885850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:07.496655 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.496620    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66279707-0137-4a71-87fb-da0938885850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr\" (UID: \"66279707-0137-4a71-87fb-da0938885850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:07.496830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.496704    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfbp\" (UniqueName: \"kubernetes.io/projected/66279707-0137-4a71-87fb-da0938885850-kube-api-access-rtfbp\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr\" (UID: \"66279707-0137-4a71-87fb-da0938885850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:07.497308 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.497287    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66279707-0137-4a71-87fb-da0938885850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr\" (UID: \"66279707-0137-4a71-87fb-da0938885850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:07.572530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.569219    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfbp\" (UniqueName: \"kubernetes.io/projected/66279707-0137-4a71-87fb-da0938885850-kube-api-access-rtfbp\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr\" (UID: \"66279707-0137-4a71-87fb-da0938885850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:07.601287 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.601260    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:07.877576 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.877451    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:50:07.877773 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.877572    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 18:50:07.877773 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.877660    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 18:50:07.926862 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.926830    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"]
Oct 31 18:50:07.927395 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.927382    5551 cpu_manager.go:414] "containerMap: removing container" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy"
Oct 31 18:50:07.927443 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.927398    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy"
Oct 31 18:50:07.927532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.927523    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a768f9-2b7c-43f0-8f36-f59966d129ae" containerName="istio-proxy"
Oct 31 18:50:07.931888 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:07.931865    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.065484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.017974    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"]
Oct 31 18:50:08.082654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.077194    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-aa4693-openshift-default-dockercfg-kvn6f\""
Oct 31 18:50:08.133596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.131670    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"]
Oct 31 18:50:08.194223 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.194111    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-socket\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.198559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.198522    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040453dc-9c8e-4741-a823-398aa8e3bc61-istiod-ca-cert\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.198829 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.198810    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-certs\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.198997 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.198983    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-podinfo\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.199134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.199120    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-data\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.199233 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.199220    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-token\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.199329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.199315    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2p7\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-kube-api-access-9d2p7\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.199490 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.199445    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-envoy\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.199668 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.199652    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-credential-socket\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.206721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.205729    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"]
Oct 31 18:50:08.206721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.205782    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"]
Oct 31 18:50:08.206721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.205852    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:08.301208 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301161    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f8622a-90a4-4c6b-898c-dcb95241545e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78\" (UID: \"46f8622a-90a4-4c6b-898c-dcb95241545e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:08.301448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301286    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-socket\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.301448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301379    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040453dc-9c8e-4741-a823-398aa8e3bc61-istiod-ca-cert\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.301448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301430    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-certs\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.301645 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301543    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-podinfo\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.301645 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301589    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phlfq\" (UniqueName: \"kubernetes.io/projected/46f8622a-90a4-4c6b-898c-dcb95241545e-kube-api-access-phlfq\") pod \"payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78\" (UID: \"46f8622a-90a4-4c6b-898c-dcb95241545e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:08.301645 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301631    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-data\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.301810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301654    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-token\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.301810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301679    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2p7\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-kube-api-access-9d2p7\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.301810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301760    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-socket\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.301810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301774    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-envoy\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.302017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301893    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-credential-socket\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.302017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301802    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-certs\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.302017 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.301956    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-data\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.302358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.302322    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-credential-socket\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.302530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.302445    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040453dc-9c8e-4741-a823-398aa8e3bc61-istiod-ca-cert\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.304416 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.304389    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-podinfo\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.304759 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.304737    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-envoy\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.378183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.378149    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-token\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.378725 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.378694    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2p7\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-kube-api-access-9d2p7\") pod \"e2e-api-a-aa4693-openshift-default-5c69d87457-b4759\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") " pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.403979 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.403850    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f8622a-90a4-4c6b-898c-dcb95241545e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78\" (UID: \"46f8622a-90a4-4c6b-898c-dcb95241545e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:08.404193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.404069    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phlfq\" (UniqueName: \"kubernetes.io/projected/46f8622a-90a4-4c6b-898c-dcb95241545e-kube-api-access-phlfq\") pod \"payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78\" (UID: \"46f8622a-90a4-4c6b-898c-dcb95241545e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:08.404846 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.404818    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f8622a-90a4-4c6b-898c-dcb95241545e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78\" (UID: \"46f8622a-90a4-4c6b-898c-dcb95241545e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:08.469484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.467259    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phlfq\" (UniqueName: \"kubernetes.io/projected/46f8622a-90a4-4c6b-898c-dcb95241545e-kube-api-access-phlfq\") pod \"payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78\" (UID: \"46f8622a-90a4-4c6b-898c-dcb95241545e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:08.524730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.524689    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:08.580842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.580795    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:08.876483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.875947    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 18:50:08.876483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.876004    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 18:50:08.941616 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:08.941592    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"]
Oct 31 18:50:08.942397 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:50:08.942366    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f8622a_90a4_4c6b_898c_dcb95241545e.slice/crio-ec6e1beb6bef612c696cabbaff08814625ba701fd3f8c7a0ab3d364afa79fcb5 WatchSource:0}: Error finding container ec6e1beb6bef612c696cabbaff08814625ba701fd3f8c7a0ab3d364afa79fcb5: Status 404 returned error can't find the container with id ec6e1beb6bef612c696cabbaff08814625ba701fd3f8c7a0ab3d364afa79fcb5
Oct 31 18:50:09.046640 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.046613    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"]
Oct 31 18:50:09.048175 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:50:09.048144    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040453dc_9c8e_4741_a823_398aa8e3bc61.slice/crio-67d3374c227c157209d1f17a241623a393ccc32d628b3d7f6325832a3cff3f97 WatchSource:0}: Error finding container 67d3374c227c157209d1f17a241623a393ccc32d628b3d7f6325832a3cff3f97: Status 404 returned error can't find the container with id 67d3374c227c157209d1f17a241623a393ccc32d628b3d7f6325832a3cff3f97
Oct 31 18:50:09.083481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.072138    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:09.083481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.074565    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:09.083481 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.074611    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:09.213253 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.213171    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" event={"ID":"040453dc-9c8e-4741-a823-398aa8e3bc61","Type":"ContainerStarted","Data":"208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad"}
Oct 31 18:50:09.213253 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.213238    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" event={"ID":"040453dc-9c8e-4741-a823-398aa8e3bc61","Type":"ContainerStarted","Data":"67d3374c227c157209d1f17a241623a393ccc32d628b3d7f6325832a3cff3f97"}
Oct 31 18:50:09.215033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.215006    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78" event={"ID":"46f8622a-90a4-4c6b-898c-dcb95241545e","Type":"ContainerStarted","Data":"ec6e1beb6bef612c696cabbaff08814625ba701fd3f8c7a0ab3d364afa79fcb5"}
Oct 31 18:50:09.216118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.216100    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr" event={"ID":"66279707-0137-4a71-87fb-da0938885850","Type":"ContainerStarted","Data":"2478bd7dd9c2212dae32a6fb1b2700b03a4bb6b464e79894ecc418d89f533efe"}
Oct 31 18:50:09.334073 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.334019    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" podStartSLOduration=2.334006284 podStartE2EDuration="2.334006284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:09.331094688 +0000 UTC m=+1712.562332978" watchObservedRunningTime="2026-10-31 18:50:09.334006284 +0000 UTC m=+1712.565244574"
Oct 31 18:50:09.582527 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.581547    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:09.878577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.876980    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 18:50:09.878577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:09.877044    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 18:50:10.187183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:10.186811    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:50:10.187183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:10.186873    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:50:10.589573 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:10.589487    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:10.617215 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:10.616935    5551 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:50:10.617215 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:10.616984    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:50:10.877121 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:10.877085    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 18:50:10.877324 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:10.877161    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 18:50:11.236565 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:11.235697    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:11.237031 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:11.236967    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:50:11.878485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:11.876870    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 18:50:11.878485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:11.876929    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 18:50:12.264777 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.264581    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr" event={"ID":"66279707-0137-4a71-87fb-da0938885850","Type":"ContainerStarted","Data":"1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0"}
Oct 31 18:50:12.264777 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.264736    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:12.266134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.266109    5551 generic.go:356] "Generic (PLEG): container finished" podID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerID="8897bb8ec178a8df99409821775ec5873dd3afcc7d5aaba7292df1f6c4aaec22" exitCode=0
Oct 31 18:50:12.266257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.266238    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" event={"ID":"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12","Type":"ContainerDied","Data":"8897bb8ec178a8df99409821775ec5873dd3afcc7d5aaba7292df1f6c4aaec22"}
Oct 31 18:50:12.266323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.266287    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk" event={"ID":"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12","Type":"ContainerDied","Data":"dfacc4a8de25b61e8d4bca9993876f19e783b1c4567bd7d3c19346957368566e"}
Oct 31 18:50:12.266323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.266313    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfacc4a8de25b61e8d4bca9993876f19e783b1c4567bd7d3c19346957368566e"
Oct 31 18:50:12.267754 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.267561    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:50:12.268898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.268716    5551 generic.go:356] "Generic (PLEG): container finished" podID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerID="e7fb9169eb0dd25cb5e2fca2fd2497747e742175dd9694090fabe6cd5fbb05fa" exitCode=0
Oct 31 18:50:12.268898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.268780    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" event={"ID":"46d1768a-b31b-40e3-b555-6e12d0542b5a","Type":"ContainerDied","Data":"e7fb9169eb0dd25cb5e2fca2fd2497747e742175dd9694090fabe6cd5fbb05fa"}
Oct 31 18:50:12.268898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.268814    5551 scope.go:122] "RemoveContainer" containerID="7958d5da7886ce2650e8603a0ebbdabd309dbe74fab04f85b015c3a6856569de"
Oct 31 18:50:12.270728 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.270254    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:50:12.271211 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.271191    5551 generic.go:356] "Generic (PLEG): container finished" podID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerID="d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a" exitCode=0
Oct 31 18:50:12.271406 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.271330    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" event={"ID":"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323","Type":"ContainerDied","Data":"d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a"}
Oct 31 18:50:12.271406 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.271375    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn" event={"ID":"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323","Type":"ContainerDied","Data":"d7c6012d402a67221cea36ad846bcf62b958c9451d9e48201f6820ab828504db"}
Oct 31 18:50:12.273786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.273767    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:50:12.284129 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.284049    5551 scope.go:122] "RemoveContainer" containerID="d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a"
Oct 31 18:50:12.297152 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.297127    5551 scope.go:122] "RemoveContainer" containerID="d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a"
Oct 31 18:50:12.297597 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:50:12.297574    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a\": container with ID starting with d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a not found: ID does not exist" containerID="d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a"
Oct 31 18:50:12.297776 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.297750    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a"} err="failed to get container status \"d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a\": rpc error: code = NotFound desc = could not find container \"d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a\": container with ID starting with d73a5d3e427f536cc2e3345e6934e3452a9413a81c891c84d7ba6e24f55f1d4a not found: ID does not exist"
Oct 31 18:50:12.372133 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372096    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-data\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.372307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372161    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-socket\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.372307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372182    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-certs\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.372307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372219    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-socket\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.372307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372239    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-envoy\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.372307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372265    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-credential-socket\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372311    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-data\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372328    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-credential-socket\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372367    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-podinfo\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372387    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-token\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-token\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372412    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-envoy\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372404    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-data" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372446    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-credential-socket\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372497    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-data\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372537    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-podinfo\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372562    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-podinfo\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372586    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-envoy\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372604    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-credential-socket" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.372649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372611    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-socket\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372670    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istiod-ca-cert\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372734    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d1768a-b31b-40e3-b555-6e12d0542b5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d1768a-b31b-40e3-b555-6e12d0542b5a-istiod-ca-cert\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372803    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-socket" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372804    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-token\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-token\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372852    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-token\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-token\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372871    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-certs\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372895    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-kube-api-access-242k5\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-kube-api-access-242k5\") pod \"46d1768a-b31b-40e3-b555-6e12d0542b5a\" (UID: \"46d1768a-b31b-40e3-b555-6e12d0542b5a\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372931    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-certs\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372953    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-kube-api-access-2fn8p\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-kube-api-access-2fn8p\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.372985    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-kube-api-access-fprvp\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-kube-api-access-fprvp\") pod \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\" (UID: \"0f06fe01-12f2-4d5e-add2-b30b9aeaeb12\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.373028    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istiod-ca-cert\") pod \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\" (UID: \"9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323\") "
Oct 31 18:50:12.373351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.373299    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-certs" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.374060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.373830    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.374060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.373849    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.374060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.373863    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.374060 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.373878    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.374912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.374725    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-credential-socket" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.375524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.375103    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-data" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.376350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.376155    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-socket" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.376988 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.376830    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-socket" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.378881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.377305    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-podinfo" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:50:12.378881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.377950    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-token" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.378881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.378000    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-token" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.378881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.378001    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-podinfo" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:50:12.380182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.379614    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-certs" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.381503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.380599    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-envoy" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.381503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.380901    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istiod-ca-cert" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:12.381503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.381246    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d1768a-b31b-40e3-b555-6e12d0542b5a-istiod-ca-cert" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:12.381503 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.381235    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr" podStartSLOduration=1.875604265 podStartE2EDuration="5.381222223s" totalImagesPullingTime="3.505617958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:12.379778191 +0000 UTC m=+1715.611016517" watchObservedRunningTime="2026-10-31 18:50:12.381222223 +0000 UTC m=+1715.612460530"
Oct 31 18:50:12.381747 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.381695    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-credential-socket" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.381966 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.381944    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-kube-api-access-242k5" pod "46d1768a-b31b-40e3-b555-6e12d0542b5a" (UID: "46d1768a-b31b-40e3-b555-6e12d0542b5a"). InnerVolumeSpecName "kube-api-access-242k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.382256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.382237    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istiod-ca-cert" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:12.382445 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.382427    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-data" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.382629 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.382612    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-certs" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.384027 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.384001    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-podinfo" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:50:12.384136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.384105    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-envoy" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.384291 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.384271    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-token" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.384506 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.384484    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-envoy" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.384604 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.384512    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-kube-api-access-2fn8p" pod "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" (UID: "9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323"). InnerVolumeSpecName "kube-api-access-2fn8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.384701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.384680    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-kube-api-access-fprvp" pod "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" (UID: "0f06fe01-12f2-4d5e-add2-b30b9aeaeb12"). InnerVolumeSpecName "kube-api-access-fprvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.475727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475595    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.475727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475630    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.475727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475645    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.475727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475661    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.475727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475674    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.475727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475690    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.475727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475704    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.475727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475719    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475735    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475749    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d1768a-b31b-40e3-b555-6e12d0542b5a-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475762    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475776    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475789    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d1768a-b31b-40e3-b555-6e12d0542b5a-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475803    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475816    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475831    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-242k5\" (UniqueName: \"kubernetes.io/projected/46d1768a-b31b-40e3-b555-6e12d0542b5a-kube-api-access-242k5\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475847    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475863    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fn8p\" (UniqueName: \"kubernetes.io/projected/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-kube-api-access-2fn8p\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475879    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fprvp\" (UniqueName: \"kubernetes.io/projected/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-kube-api-access-fprvp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475894    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475908    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475922    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d1768a-b31b-40e3-b555-6e12d0542b5a-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.476413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.475935    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.881532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.881502    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:50:12.882061 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.882037    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:50:12.883632 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:12.883608    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:50:13.281234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.281146    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph" event={"ID":"46d1768a-b31b-40e3-b555-6e12d0542b5a","Type":"ContainerDied","Data":"59b6c38d96ff431165d07926fb460828f6a7960ac8da0bbddb6f802b1750eaa3"}
Oct 31 18:50:13.281234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.281219    5551 scope.go:122] "RemoveContainer" containerID="e7fb9169eb0dd25cb5e2fca2fd2497747e742175dd9694090fabe6cd5fbb05fa"
Oct 31 18:50:13.281669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.281143    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"
Oct 31 18:50:13.282762 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.282740    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"
Oct 31 18:50:13.285877 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.285286    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"
Oct 31 18:50:13.286286 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.286254    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78" event={"ID":"46f8622a-90a4-4c6b-898c-dcb95241545e","Type":"ContainerStarted","Data":"a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95"}
Oct 31 18:50:13.286383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.286327    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:13.520290 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.520222    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78" podStartSLOduration=3.357590496 podStartE2EDuration="6.520203245s" totalImagesPullingTime="3.162612749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:13.519067998 +0000 UTC m=+1716.750306290" watchObservedRunningTime="2026-10-31 18:50:13.520203245 +0000 UTC m=+1716.751441536"
Oct 31 18:50:13.638205 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.638167    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"]
Oct 31 18:50:13.651997 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.651962    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-frxwn"]
Oct 31 18:50:13.757951 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.757921    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"]
Oct 31 18:50:13.762542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.762514    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-9msph"]
Oct 31 18:50:13.871155 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.871115    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"]
Oct 31 18:50:13.875310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:13.875285    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-t5vwk"]
Oct 31 18:50:14.909139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:14.909108    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" path="/var/lib/kubelet/pods/0f06fe01-12f2-4d5e-add2-b30b9aeaeb12/volumes"
Oct 31 18:50:14.909601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:14.909585    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" path="/var/lib/kubelet/pods/46d1768a-b31b-40e3-b555-6e12d0542b5a/volumes"
Oct 31 18:50:14.910038 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:14.910025    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" path="/var/lib/kubelet/pods/9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323/volumes"
Oct 31 18:50:16.605480 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:50:16.605426    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:50:18.675854 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.675817    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"]
Oct 31 18:50:18.676685 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676664    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy"
Oct 31 18:50:18.676744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676688    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy"
Oct 31 18:50:18.676779 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676752    5551 cpu_manager.go:414] "containerMap: removing container" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy"
Oct 31 18:50:18.676779 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676761    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy"
Oct 31 18:50:18.676845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676790    5551 cpu_manager.go:414] "containerMap: removing container" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy"
Oct 31 18:50:18.676845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676798    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy"
Oct 31 18:50:18.676927 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676916    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy"
Oct 31 18:50:18.676964 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676934    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9ddfe3-e3c0-4f86-b93c-b9ba98adf323" containerName="istio-proxy"
Oct 31 18:50:18.676964 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.676948    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f06fe01-12f2-4d5e-add2-b30b9aeaeb12" containerName="istio-proxy"
Oct 31 18:50:18.682353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.682331    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.739159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.739131    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-100faf-openshift-default-dockercfg-xvv8w\""
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753173    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-podinfo\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753210    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-data\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753295    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-credential-socket\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753328    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d091f-e6cc-4694-ae3f-783874d1e09e-istiod-ca-cert\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753409    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-token\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753436    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-certs\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753485    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-socket\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753534    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69lp\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-kube-api-access-h69lp\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.755549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.753558    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-envoy\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.794892 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.794854    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"]
Oct 31 18:50:18.854501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.854447    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-credential-socket\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.854790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.854761    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d091f-e6cc-4694-ae3f-783874d1e09e-istiod-ca-cert\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.855067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.855046    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-token\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.855222 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.855201    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-certs\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.855350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.855337    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-socket\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.855499 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.855485    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69lp\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-kube-api-access-h69lp\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.855629 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.855614    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-envoy\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.855763 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.855750    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-podinfo\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.855858 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.855846    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-data\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.856315 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.856284    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-certs\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.856480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.856441    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-data\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.856591 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.855533    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d091f-e6cc-4694-ae3f-783874d1e09e-istiod-ca-cert\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.856682 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.856597    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-socket\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.856768 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.854856    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-credential-socket\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.864659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.864603    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-envoy\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.869246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.869224    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-podinfo\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.912497 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.912434    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69lp\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-kube-api-access-h69lp\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:18.912825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:18.912803    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-token\") pod \"e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") " pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:19.005491 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:19.005289    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:19.462195 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:19.462166    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"]
Oct 31 18:50:19.464499 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:50:19.464467    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604d091f_e6cc_4694_ae3f_783874d1e09e.slice/crio-f5bc4df0fb12384fd2ada2c010a4f16af4f670a6f1e2625d5ba6bf91a4256471 WatchSource:0}: Error finding container f5bc4df0fb12384fd2ada2c010a4f16af4f670a6f1e2625d5ba6bf91a4256471: Status 404 returned error can't find the container with id f5bc4df0fb12384fd2ada2c010a4f16af4f670a6f1e2625d5ba6bf91a4256471
Oct 31 18:50:19.467728 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:19.467683    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:19.467821 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:19.467776    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:19.467862 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:19.467822    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:20.335698 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:20.335655    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" event={"ID":"604d091f-e6cc-4694-ae3f-783874d1e09e","Type":"ContainerStarted","Data":"952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9"}
Oct 31 18:50:20.336113 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:20.335723    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" event={"ID":"604d091f-e6cc-4694-ae3f-783874d1e09e","Type":"ContainerStarted","Data":"f5bc4df0fb12384fd2ada2c010a4f16af4f670a6f1e2625d5ba6bf91a4256471"}
Oct 31 18:50:20.506433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:20.506383    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" podStartSLOduration=2.506369529 podStartE2EDuration="2.506369529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:20.504492896 +0000 UTC m=+1723.735731188" watchObservedRunningTime="2026-10-31 18:50:20.506369529 +0000 UTC m=+1723.737607820"
Oct 31 18:50:21.007025 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:21.006991    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:21.013551 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:21.013523    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:21.343171 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:21.343082    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:21.343964 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:21.343942    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:50:23.288558 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:23.288517    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:50:24.304672 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:24.304629    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:50:30.501323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:30.501255    5551 generic.go:356] "Generic (PLEG): container finished" podID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerID="6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476" exitCode=0
Oct 31 18:50:30.501747 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:30.501337    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" event={"ID":"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705","Type":"ContainerDied","Data":"6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476"}
Oct 31 18:50:30.501856 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:30.501838    5551 scope.go:122] "RemoveContainer" containerID="6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476"
Oct 31 18:50:30.503402 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:30.503370    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:30.503494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:30.503439    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:30.503540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:30.503504    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:50:31.510650 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:31.510618    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" event={"ID":"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705","Type":"ContainerStarted","Data":"7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c"}
Oct 31 18:50:31.634988 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:31.634960    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:50:31.635165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:31.635131    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 18:50:31.635213 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:31.635183    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 18:50:32.635624 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:32.635588    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 18:50:32.636021 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:32.635654    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 18:50:33.635350 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:33.635314    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 18:50:33.635566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:33.635368    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 18:50:34.635378 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:34.635341    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 18:50:34.635962 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:34.635409    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 18:50:35.635520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:35.635450    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 18:50:35.635906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:35.635537    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 18:50:36.635436 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:36.635396    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 18:50:36.635671 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:36.635478    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 18:50:37.292975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:37.292939    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:50:37.639818 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:37.639792    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:50:37.640836 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:50:37.640817    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:50:46.604668 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:50:46.604629    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:51:03.548371 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.548331    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"]
Oct 31 18:51:03.549193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.549170    5551 cpu_manager.go:414] "containerMap: removing container" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy"
Oct 31 18:51:03.549301 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.549196    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy"
Oct 31 18:51:03.549408 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.549393    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d1768a-b31b-40e3-b555-6e12d0542b5a" containerName="istio-proxy"
Oct 31 18:51:03.552792 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.552771    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:03.608170 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.608126    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"]
Oct 31 18:51:03.621699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.621669    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c9021-523b-4b02-a8eb-21cb2161156d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs\" (UID: \"ea8c9021-523b-4b02-a8eb-21cb2161156d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:03.621856 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.621836    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g45s\" (UniqueName: \"kubernetes.io/projected/ea8c9021-523b-4b02-a8eb-21cb2161156d-kube-api-access-8g45s\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs\" (UID: \"ea8c9021-523b-4b02-a8eb-21cb2161156d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:03.653571 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.653540    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5dc7b6\""
Oct 31 18:51:03.653703 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.653617    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5dc7b6-dockercfg-sq8bf\""
Oct 31 18:51:03.723277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.723239    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c9021-523b-4b02-a8eb-21cb2161156d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs\" (UID: \"ea8c9021-523b-4b02-a8eb-21cb2161156d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:03.723495 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.723363    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g45s\" (UniqueName: \"kubernetes.io/projected/ea8c9021-523b-4b02-a8eb-21cb2161156d-kube-api-access-8g45s\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs\" (UID: \"ea8c9021-523b-4b02-a8eb-21cb2161156d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:03.724086 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.724063    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c9021-523b-4b02-a8eb-21cb2161156d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs\" (UID: \"ea8c9021-523b-4b02-a8eb-21cb2161156d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:03.780259 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.780230    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g45s\" (UniqueName: \"kubernetes.io/projected/ea8c9021-523b-4b02-a8eb-21cb2161156d-kube-api-access-8g45s\") pod \"payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs\" (UID: \"ea8c9021-523b-4b02-a8eb-21cb2161156d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:03.866951 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:03.866864    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:04.243975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.243947    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"]
Oct 31 18:51:04.245046 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:51:04.245021    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8c9021_523b_4b02_a8eb_21cb2161156d.slice/crio-b2a9feb83f0a34115b383268405ace9ae88529b47752fb858f476d8a5e1d4bce WatchSource:0}: Error finding container b2a9feb83f0a34115b383268405ace9ae88529b47752fb858f476d8a5e1d4bce: Status 404 returned error can't find the container with id b2a9feb83f0a34115b383268405ace9ae88529b47752fb858f476d8a5e1d4bce
Oct 31 18:51:04.331187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.331162    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"]
Oct 31 18:51:04.335219 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.335200    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:04.398906 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.398869    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"]
Oct 31 18:51:04.431690 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.431655    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg\" (UID: \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:04.431845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.431701    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zws5c\" (UniqueName: \"kubernetes.io/projected/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-kube-api-access-zws5c\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg\" (UID: \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:04.532993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.532895    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg\" (UID: \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:04.532993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.532954    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zws5c\" (UniqueName: \"kubernetes.io/projected/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-kube-api-access-zws5c\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg\" (UID: \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:04.533629 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.533609    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg\" (UID: \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:04.589677 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.589649    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zws5c\" (UniqueName: \"kubernetes.io/projected/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-kube-api-access-zws5c\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg\" (UID: \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:04.651238 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.651207    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:04.769744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.769704    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs" event={"ID":"ea8c9021-523b-4b02-a8eb-21cb2161156d","Type":"ContainerStarted","Data":"4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842"}
Oct 31 18:51:04.769923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.769772    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs" event={"ID":"ea8c9021-523b-4b02-a8eb-21cb2161156d","Type":"ContainerStarted","Data":"b2a9feb83f0a34115b383268405ace9ae88529b47752fb858f476d8a5e1d4bce"}
Oct 31 18:51:04.769923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.769819    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:04.882878 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:04.882826    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs" podStartSLOduration=1.882812501 podStartE2EDuration="1.882812501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:04.881578693 +0000 UTC m=+1768.112816988" watchObservedRunningTime="2026-10-31 18:51:04.882812501 +0000 UTC m=+1768.114050791"
Oct 31 18:51:05.029861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.029826    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"]
Oct 31 18:51:05.031760 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:51:05.031728    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041cc576_f4f6_4acd_b95b_0c800cf0c6b3.slice/crio-707f9d6337e9ef17370ab61d1c42bd630dc0fd659a33d051657514307d1df92a WatchSource:0}: Error finding container 707f9d6337e9ef17370ab61d1c42bd630dc0fd659a33d051657514307d1df92a: Status 404 returned error can't find the container with id 707f9d6337e9ef17370ab61d1c42bd630dc0fd659a33d051657514307d1df92a
Oct 31 18:51:05.041517 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.037838    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"]
Oct 31 18:51:05.043938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.043917    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.095843 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.095640    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-aa4693-serving-cert\""
Oct 31 18:51:05.097884 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.097860    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"]
Oct 31 18:51:05.140261 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.140156    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-metrics-certs\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.140427 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.140384    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrbb\" (UniqueName: \"kubernetes.io/projected/02eebe21-8f57-4df3-a107-122eb0d46678-kube-api-access-dqrbb\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.140542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.140522    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eebe21-8f57-4df3-a107-122eb0d46678-openshift-service-ca\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.140603 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.140552    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.241540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.241510    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-metrics-certs\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.241710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.241639    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrbb\" (UniqueName: \"kubernetes.io/projected/02eebe21-8f57-4df3-a107-122eb0d46678-kube-api-access-dqrbb\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.241710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.241695    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eebe21-8f57-4df3-a107-122eb0d46678-openshift-service-ca\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.241792 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.241716    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.241850 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:51:05.241836    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-serving-cert: secret "maas-api-e2e-api-a-aa4693-serving-cert" not found
Oct 31 18:51:05.241903 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:51:05.241893    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls podName:02eebe21-8f57-4df3-a107-122eb0d46678 nodeName:}" failed. No retries permitted until 2026-10-31 18:51:05.741872409 +0000 UTC m=+1768.973110682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls") pod "maas-api-e2e-api-a-aa4693-66cd556984-qnlvk" (UID: "02eebe21-8f57-4df3-a107-122eb0d46678") : secret "maas-api-e2e-api-a-aa4693-serving-cert" not found
Oct 31 18:51:05.242334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.242314    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eebe21-8f57-4df3-a107-122eb0d46678-openshift-service-ca\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.244179 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.244155    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-metrics-certs\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.327164 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.324673    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrbb\" (UniqueName: \"kubernetes.io/projected/02eebe21-8f57-4df3-a107-122eb0d46678-kube-api-access-dqrbb\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.747861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.747825    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.750473 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.750431    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls\") pod \"maas-api-e2e-api-a-aa4693-66cd556984-qnlvk\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:05.778131 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.778096    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg" event={"ID":"041cc576-f4f6-4acd-b95b-0c800cf0c6b3","Type":"ContainerStarted","Data":"9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95"}
Oct 31 18:51:05.778260 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.778158    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg" event={"ID":"041cc576-f4f6-4acd-b95b-0c800cf0c6b3","Type":"ContainerStarted","Data":"707f9d6337e9ef17370ab61d1c42bd630dc0fd659a33d051657514307d1df92a"}
Oct 31 18:51:05.778353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.778338    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:05.892791 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.892735    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg" podStartSLOduration=1.892720559 podStartE2EDuration="1.892720559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:05.890371903 +0000 UTC m=+1769.121610195" watchObservedRunningTime="2026-10-31 18:51:05.892720559 +0000 UTC m=+1769.123958848"
Oct 31 18:51:05.985515 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:05.985468    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:06.399337 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:06.399306    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"]
Oct 31 18:51:06.400391 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:51:06.400361    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eebe21_8f57_4df3_a107_122eb0d46678.slice/crio-cf0355aaad65954bd1c050db9591bdaf3b98f5c320bac5e75ee0cca0b9850f3b WatchSource:0}: Error finding container cf0355aaad65954bd1c050db9591bdaf3b98f5c320bac5e75ee0cca0b9850f3b: Status 404 returned error can't find the container with id cf0355aaad65954bd1c050db9591bdaf3b98f5c320bac5e75ee0cca0b9850f3b
Oct 31 18:51:06.787663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:06.787569    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk" event={"ID":"02eebe21-8f57-4df3-a107-122eb0d46678","Type":"ContainerStarted","Data":"cf0355aaad65954bd1c050db9591bdaf3b98f5c320bac5e75ee0cca0b9850f3b"}
Oct 31 18:51:07.796259 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:07.796217    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk" event={"ID":"02eebe21-8f57-4df3-a107-122eb0d46678","Type":"ContainerStarted","Data":"bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187"}
Oct 31 18:51:07.796686 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:07.796439    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:07.912207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:07.912158    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk" podStartSLOduration=2.987343824 podStartE2EDuration="3.912144954s" totalImagesPullingTime="924.80113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:07.909843657 +0000 UTC m=+1771.141081949" watchObservedRunningTime="2026-10-31 18:51:07.912144954 +0000 UTC m=+1771.143383240"
Oct 31 18:51:13.809640 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:13.809608    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:51:15.780531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:15.780492    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:51:16.606097 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:51:16.606056    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:51:16.790649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:16.790621    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:51:38.935057 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:38.935023    5551 scope.go:122] "RemoveContainer" containerID="df95fd8bbd2e2dfd5ecb75cb0ecb423f64c13d300548aa2ad1dce80dc8ad1131"
Oct 31 18:51:40.013348 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:40.013314    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:51:40.045241 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:40.045213    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:51:41.072981 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:41.072908    5551 generic.go:356] "Generic (PLEG): container finished" podID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerID="208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad" exitCode=0
Oct 31 18:51:41.073515 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:41.072985    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" event={"ID":"040453dc-9c8e-4741-a823-398aa8e3bc61","Type":"ContainerDied","Data":"208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad"}
Oct 31 18:51:41.073515 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:41.073381    5551 scope.go:122] "RemoveContainer" containerID="208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad"
Oct 31 18:51:41.075050 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:41.075023    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:51:41.075171 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:41.075089    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:51:41.075171 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:41.075123    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:51:41.235888 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:41.235854    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:51:42.090660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:42.090625    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" event={"ID":"040453dc-9c8e-4741-a823-398aa8e3bc61","Type":"ContainerStarted","Data":"38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582"}
Oct 31 18:51:42.581873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:42.581772    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:51:42.583559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:42.583517    5551 patch_prober.go:29] interesting pod/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 18:51:42.583717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:42.583589    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 18:51:43.581398 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:43.581346    5551 patch_prober.go:29] interesting pod/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 18:51:43.581865 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:43.581427    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 18:51:44.581949 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:44.581911    5551 patch_prober.go:29] interesting pod/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 18:51:44.582358 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:44.581967    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 18:51:45.581875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:45.581834    5551 patch_prober.go:29] interesting pod/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 18:51:45.582059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:45.581907    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 18:51:46.581973 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:46.581937    5551 patch_prober.go:29] interesting pod/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 18:51:46.582145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:46.581998    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 18:51:46.607615 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:51:46.607578    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:51:47.585844 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:47.585814    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:51:47.586308 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:47.586244    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:51:47.587051 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:51:47.587030    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:52:06.354699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:06.354268    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": read tcp 10.134.0.2:51362->10.134.0.114:15021: read: connection reset by peer" start-of-body=
Oct 31 18:52:06.354699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:06.354346    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": read tcp 10.134.0.2:51362->10.134.0.114:15021: read: connection reset by peer"
Oct 31 18:52:07.318881 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:07.318848    5551 generic.go:356] "Generic (PLEG): container finished" podID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerID="952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9" exitCode=0
Oct 31 18:52:07.319082 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:07.318914    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" event={"ID":"604d091f-e6cc-4694-ae3f-783874d1e09e","Type":"ContainerDied","Data":"952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9"}
Oct 31 18:52:07.319331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:07.319303    5551 scope.go:122] "RemoveContainer" containerID="952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9"
Oct 31 18:52:07.320952 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:07.320926    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:52:07.321009 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:07.321000    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:52:07.321046 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:07.321032    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:52:08.328237 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:08.328195    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" event={"ID":"604d091f-e6cc-4694-ae3f-783874d1e09e","Type":"ContainerStarted","Data":"790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364"}
Oct 31 18:52:09.006532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:09.006493    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:52:09.006732 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:09.006670    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:52:09.006791 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:09.006727    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 18:52:10.007063 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:10.007032    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:52:10.007520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:10.007085    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 18:52:11.006801 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:11.006769    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:52:11.007012 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:11.006827    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 18:52:12.006952 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:12.006918    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:52:12.007374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:12.006976    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 18:52:13.006960 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:13.006928    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:52:13.007490 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:13.006980    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 18:52:14.011726 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:14.011685    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:52:14.012265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:14.011965    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:52:14.012904 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:14.012886    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:52:16.606987 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:52:16.606949    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:52:46.605315 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:52:46.605278    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:52:48.376353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:48.376317    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"]
Oct 31 18:52:48.376785 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:48.376556    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy" containerID="cri-o://7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227" gracePeriod=30
Oct 31 18:52:53.637613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.637587    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:52:53.695490 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.695399    5551 generic.go:356] "Generic (PLEG): container finished" podID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerID="7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227" exitCode=0
Oct 31 18:52:53.695490 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.695448    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"
Oct 31 18:52:53.695688 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.695441    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" event={"ID":"10b4dc00-057e-4d84-a25c-31ed75e4a187","Type":"ContainerDied","Data":"7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227"}
Oct 31 18:52:53.695688 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.695546    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d" event={"ID":"10b4dc00-057e-4d84-a25c-31ed75e4a187","Type":"ContainerDied","Data":"c33818ef4ddfdda324b36e7be1544251c33383dcfd3d8547ea0484febe2c12f6"}
Oct 31 18:52:53.695688 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.695583    5551 scope.go:122] "RemoveContainer" containerID="7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227"
Oct 31 18:52:53.705566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.705545    5551 scope.go:122] "RemoveContainer" containerID="7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227"
Oct 31 18:52:53.705833 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:52:53.705815    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227\": container with ID starting with 7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227 not found: ID does not exist" containerID="7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227"
Oct 31 18:52:53.705895 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.705848    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227"} err="failed to get container status \"7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227\": rpc error: code = NotFound desc = could not find container \"7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227\": container with ID starting with 7834a8fcb40d9f568265a71d0ba42a4a114582734def08f9761bd0b1b8c39227 not found: ID does not exist"
Oct 31 18:52:53.800339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800305    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-token\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-token\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.800546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800358    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-socket\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.800546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800401    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-podinfo\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.800546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800494    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-certs\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.800705 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800545    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-data\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.800705 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800575    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-envoy\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.800705 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800648    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-kube-api-access-txmfh\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-kube-api-access-txmfh\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.800867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800704    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b4dc00-057e-4d84-a25c-31ed75e4a187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b4dc00-057e-4d84-a25c-31ed75e4a187-istiod-ca-cert\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.800867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800710    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-socket" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:53.800867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800795    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-data" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:53.800867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800819    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-certs" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:53.800867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800809    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-credential-socket\") pod \"10b4dc00-057e-4d84-a25c-31ed75e4a187\" (UID: \"10b4dc00-057e-4d84-a25c-31ed75e4a187\") "
Oct 31 18:52:53.801139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.800988    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-credential-socket" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:53.801139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.801079    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b4dc00-057e-4d84-a25c-31ed75e4a187-istiod-ca-cert" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:52:53.801702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.801664    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:53.801702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.801694    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:53.801865 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.801710    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b4dc00-057e-4d84-a25c-31ed75e4a187-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:53.801865 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.801727    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:53.801865 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.801740    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:53.802918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.802897    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-token" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:52:53.803063 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.803042    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-podinfo" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:52:53.803421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.803402    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-envoy" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:53.803543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.803424    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-kube-api-access-txmfh" pod "10b4dc00-057e-4d84-a25c-31ed75e4a187" (UID: "10b4dc00-057e-4d84-a25c-31ed75e4a187"). InnerVolumeSpecName "kube-api-access-txmfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:52:53.902661 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.902631    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:53.902661 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.902659    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:53.902886 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.902672    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b4dc00-057e-4d84-a25c-31ed75e4a187-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:53.902886 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:53.902685    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txmfh\" (UniqueName: \"kubernetes.io/projected/10b4dc00-057e-4d84-a25c-31ed75e4a187-kube-api-access-txmfh\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:54.120013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:54.119978    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"]
Oct 31 18:52:54.124661 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:54.124637    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-rg96d"]
Oct 31 18:52:54.909997 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:52:54.909961    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" path="/var/lib/kubelet/pods/10b4dc00-057e-4d84-a25c-31ed75e4a187/volumes"
Oct 31 18:53:16.605168 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:16.605089    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:53:25.890265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:25.890216    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"]
Oct 31 18:53:25.892424 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:25.892392    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd" podUID="4d23327e-95c2-432f-b8ba-cdb835d739ee" containerName="maas-api" containerID="cri-o://3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c" gracePeriod=30
Oct 31 18:53:26.195224 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.195198    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:53:26.218447 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.218420    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d23327e-95c2-432f-b8ba-cdb835d739ee-kube-api-access-rs5q9\" (UniqueName: \"kubernetes.io/projected/4d23327e-95c2-432f-b8ba-cdb835d739ee-kube-api-access-rs5q9\") pod \"4d23327e-95c2-432f-b8ba-cdb835d739ee\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") "
Oct 31 18:53:26.218646 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.218626    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d23327e-95c2-432f-b8ba-cdb835d739ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d23327e-95c2-432f-b8ba-cdb835d739ee-openshift-service-ca\") pod \"4d23327e-95c2-432f-b8ba-cdb835d739ee\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") "
Oct 31 18:53:26.218694 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.218675    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-metrics-certs\") pod \"4d23327e-95c2-432f-b8ba-cdb835d739ee\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") "
Oct 31 18:53:26.218733 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.218715    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-maas-api-tls\") pod \"4d23327e-95c2-432f-b8ba-cdb835d739ee\" (UID: \"4d23327e-95c2-432f-b8ba-cdb835d739ee\") "
Oct 31 18:53:26.218934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.218913    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d23327e-95c2-432f-b8ba-cdb835d739ee-openshift-service-ca" pod "4d23327e-95c2-432f-b8ba-cdb835d739ee" (UID: "4d23327e-95c2-432f-b8ba-cdb835d739ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:26.219315 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.219228    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d23327e-95c2-432f-b8ba-cdb835d739ee-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:26.220718 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.220696    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d23327e-95c2-432f-b8ba-cdb835d739ee-kube-api-access-rs5q9" pod "4d23327e-95c2-432f-b8ba-cdb835d739ee" (UID: "4d23327e-95c2-432f-b8ba-cdb835d739ee"). InnerVolumeSpecName "kube-api-access-rs5q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:26.220806 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.220741    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-metrics-certs" pod "4d23327e-95c2-432f-b8ba-cdb835d739ee" (UID: "4d23327e-95c2-432f-b8ba-cdb835d739ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:53:26.221031 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.221005    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-maas-api-tls" pod "4d23327e-95c2-432f-b8ba-cdb835d739ee" (UID: "4d23327e-95c2-432f-b8ba-cdb835d739ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:53:26.321033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.321001    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:26.321033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.321036    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d23327e-95c2-432f-b8ba-cdb835d739ee-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:26.321263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.321052    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs5q9\" (UniqueName: \"kubernetes.io/projected/4d23327e-95c2-432f-b8ba-cdb835d739ee-kube-api-access-rs5q9\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:26.334127 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.334095    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"]
Oct 31 18:53:26.334432 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.334374    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg" podUID="041cc576-f4f6-4acd-b95b-0c800cf0c6b3" containerName="payload-processing" containerID="cri-o://9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95" gracePeriod=30
Oct 31 18:53:26.388578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.388551    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"]
Oct 31 18:53:26.388760 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.388740    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs" podUID="ea8c9021-523b-4b02-a8eb-21cb2161156d" containerName="payload-pre-processing" containerID="cri-o://4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842" gracePeriod=30
Oct 31 18:53:26.670246 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.670214    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:53:26.674677 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.674652    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:53:26.727153 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.727117    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8c9021-523b-4b02-a8eb-21cb2161156d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c9021-523b-4b02-a8eb-21cb2161156d-plugins-config-volume\") pod \"ea8c9021-523b-4b02-a8eb-21cb2161156d\" (UID: \"ea8c9021-523b-4b02-a8eb-21cb2161156d\") "
Oct 31 18:53:26.727327 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.727186    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8c9021-523b-4b02-a8eb-21cb2161156d-kube-api-access-8g45s\" (UniqueName: \"kubernetes.io/projected/ea8c9021-523b-4b02-a8eb-21cb2161156d-kube-api-access-8g45s\") pod \"ea8c9021-523b-4b02-a8eb-21cb2161156d\" (UID: \"ea8c9021-523b-4b02-a8eb-21cb2161156d\") "
Oct 31 18:53:26.727327 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.727248    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-kube-api-access-zws5c\" (UniqueName: \"kubernetes.io/projected/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-kube-api-access-zws5c\") pod \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\" (UID: \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\") "
Oct 31 18:53:26.727501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.727415    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-plugins-config-volume\") pod \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\" (UID: \"041cc576-f4f6-4acd-b95b-0c800cf0c6b3\") "
Oct 31 18:53:26.727596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.727567    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8c9021-523b-4b02-a8eb-21cb2161156d-plugins-config-volume" pod "ea8c9021-523b-4b02-a8eb-21cb2161156d" (UID: "ea8c9021-523b-4b02-a8eb-21cb2161156d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:26.727841 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.727819    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-plugins-config-volume" pod "041cc576-f4f6-4acd-b95b-0c800cf0c6b3" (UID: "041cc576-f4f6-4acd-b95b-0c800cf0c6b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:26.727944 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.727880    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8c9021-523b-4b02-a8eb-21cb2161156d-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:26.729607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.729587    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-kube-api-access-zws5c" pod "041cc576-f4f6-4acd-b95b-0c800cf0c6b3" (UID: "041cc576-f4f6-4acd-b95b-0c800cf0c6b3"). InnerVolumeSpecName "kube-api-access-zws5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:26.729898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.729884    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8c9021-523b-4b02-a8eb-21cb2161156d-kube-api-access-8g45s" pod "ea8c9021-523b-4b02-a8eb-21cb2161156d" (UID: "ea8c9021-523b-4b02-a8eb-21cb2161156d"). InnerVolumeSpecName "kube-api-access-8g45s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:26.828874 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.828789    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g45s\" (UniqueName: \"kubernetes.io/projected/ea8c9021-523b-4b02-a8eb-21cb2161156d-kube-api-access-8g45s\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:26.828874 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.828819    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zws5c\" (UniqueName: \"kubernetes.io/projected/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-kube-api-access-zws5c\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:26.828874 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:26.828829    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041cc576-f4f6-4acd-b95b-0c800cf0c6b3-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:27.014309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.013925    5551 generic.go:356] "Generic (PLEG): container finished" podID="041cc576-f4f6-4acd-b95b-0c800cf0c6b3" containerID="9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95" exitCode=0
Oct 31 18:53:27.014309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.014115    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg" event={"ID":"041cc576-f4f6-4acd-b95b-0c800cf0c6b3","Type":"ContainerDied","Data":"9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95"}
Oct 31 18:53:27.014309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.014168    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg" event={"ID":"041cc576-f4f6-4acd-b95b-0c800cf0c6b3","Type":"ContainerDied","Data":"707f9d6337e9ef17370ab61d1c42bd630dc0fd659a33d051657514307d1df92a"}
Oct 31 18:53:27.014309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.014197    5551 scope.go:122] "RemoveContainer" containerID="9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95"
Oct 31 18:53:27.014309 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.014262    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"
Oct 31 18:53:27.018172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.018124    5551 generic.go:356] "Generic (PLEG): container finished" podID="ea8c9021-523b-4b02-a8eb-21cb2161156d" containerID="4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842" exitCode=0
Oct 31 18:53:27.018380 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.018348    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs" event={"ID":"ea8c9021-523b-4b02-a8eb-21cb2161156d","Type":"ContainerDied","Data":"4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842"}
Oct 31 18:53:27.018722 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.018618    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs" event={"ID":"ea8c9021-523b-4b02-a8eb-21cb2161156d","Type":"ContainerDied","Data":"b2a9feb83f0a34115b383268405ace9ae88529b47752fb858f476d8a5e1d4bce"}
Oct 31 18:53:27.018996 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.018927    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"
Oct 31 18:53:27.024594 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.024578    5551 generic.go:356] "Generic (PLEG): container finished" podID="4d23327e-95c2-432f-b8ba-cdb835d739ee" containerID="3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c" exitCode=0
Oct 31 18:53:27.024730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.024719    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd" event={"ID":"4d23327e-95c2-432f-b8ba-cdb835d739ee","Type":"ContainerDied","Data":"3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c"}
Oct 31 18:53:27.024810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.024801    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd" event={"ID":"4d23327e-95c2-432f-b8ba-cdb835d739ee","Type":"ContainerDied","Data":"9c9100d46b04b55bb722ef5b8e075ec064c750289c40b0b358579bcb3ad8b660"}
Oct 31 18:53:27.024920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.024911    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"
Oct 31 18:53:27.035819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.035511    5551 scope.go:122] "RemoveContainer" containerID="9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95"
Oct 31 18:53:27.036024 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:27.035997    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95\": container with ID starting with 9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95 not found: ID does not exist" containerID="9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95"
Oct 31 18:53:27.036075 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.036039    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95"} err="failed to get container status \"9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95\": rpc error: code = NotFound desc = could not find container \"9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95\": container with ID starting with 9d003c4898c2b236d990f6518c917601a759309043fea677f56fd661f3053c95 not found: ID does not exist"
Oct 31 18:53:27.036075 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.036066    5551 scope.go:122] "RemoveContainer" containerID="4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842"
Oct 31 18:53:27.047532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.047510    5551 scope.go:122] "RemoveContainer" containerID="4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842"
Oct 31 18:53:27.047873 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:27.047850    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842\": container with ID starting with 4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842 not found: ID does not exist" containerID="4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842"
Oct 31 18:53:27.047968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.047880    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842"} err="failed to get container status \"4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842\": rpc error: code = NotFound desc = could not find container \"4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842\": container with ID starting with 4121e73931f838f273e255874860fe73713fb687089777c88931f66b37978842 not found: ID does not exist"
Oct 31 18:53:27.047968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.047899    5551 scope.go:122] "RemoveContainer" containerID="3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c"
Oct 31 18:53:27.060925 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.060901    5551 scope.go:122] "RemoveContainer" containerID="3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c"
Oct 31 18:53:27.061197 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:27.061177    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c\": container with ID starting with 3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c not found: ID does not exist" containerID="3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c"
Oct 31 18:53:27.061261 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.061204    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c"} err="failed to get container status \"3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c\": rpc error: code = NotFound desc = could not find container \"3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c\": container with ID starting with 3adde80ada73f498455f3bf8093c1ccab0a601d48d800f2ac774b21494789e3c not found: ID does not exist"
Oct 31 18:53:27.170501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.170449    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"]
Oct 31 18:53:27.175032 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.175007    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78b9649448-qhvtg"]
Oct 31 18:53:27.278912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.278884    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"]
Oct 31 18:53:27.282893 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.282869    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dc7b6-d95b4d97f-bzbqs"]
Oct 31 18:53:27.391172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.391129    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"]
Oct 31 18:53:27.396529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:27.396490    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5dc7b6-77f77bf7f8-s5nrd"]
Oct 31 18:53:28.909334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:28.909298    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="041cc576-f4f6-4acd-b95b-0c800cf0c6b3" path="/var/lib/kubelet/pods/041cc576-f4f6-4acd-b95b-0c800cf0c6b3/volumes"
Oct 31 18:53:28.909791 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:28.909776    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d23327e-95c2-432f-b8ba-cdb835d739ee" path="/var/lib/kubelet/pods/4d23327e-95c2-432f-b8ba-cdb835d739ee/volumes"
Oct 31 18:53:28.910158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:28.910143    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8c9021-523b-4b02-a8eb-21cb2161156d" path="/var/lib/kubelet/pods/ea8c9021-523b-4b02-a8eb-21cb2161156d/volumes"
Oct 31 18:53:31.457994 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:31.457959    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"]
Oct 31 18:53:31.458394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:31.458261    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy" containerID="cri-o://1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3" gracePeriod=30
Oct 31 18:53:36.732583 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.732520    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:53:36.773832 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.773771    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-certs\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.773906    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-credential-socket\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.773959    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1563baac-7e8c-45f1-aa23-a6eb2430341d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563baac-7e8c-45f1-aa23-a6eb2430341d-istiod-ca-cert\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774033 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774015    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-certs" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:36.774165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774110    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-envoy\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774189    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-socket\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774214    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-credential-socket" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:36.774320 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774227    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-podinfo\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774334    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-kube-api-access-jblqk\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-kube-api-access-jblqk\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774377 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774369    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-token\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-token\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774415    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-data\") pod \"1563baac-7e8c-45f1-aa23-a6eb2430341d\" (UID: \"1563baac-7e8c-45f1-aa23-a6eb2430341d\") "
Oct 31 18:53:36.774849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774487    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-socket" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:36.774849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774521    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1563baac-7e8c-45f1-aa23-a6eb2430341d-istiod-ca-cert" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:36.774849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.774810    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-data" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:36.775277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.775226    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:36.775277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.775249    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:36.775277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.775265    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:36.775451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.775282    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:36.775451 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.775297    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563baac-7e8c-45f1-aa23-a6eb2430341d-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:36.776842 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.776816    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-token" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:36.776958 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.776940    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-podinfo" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:53:36.777114 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.777089    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-envoy" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:36.777345 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.777324    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-kube-api-access-jblqk" pod "1563baac-7e8c-45f1-aa23-a6eb2430341d" (UID: "1563baac-7e8c-45f1-aa23-a6eb2430341d"). InnerVolumeSpecName "kube-api-access-jblqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:36.876172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.876141    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:36.876172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.876170    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:36.876392 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.876180    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jblqk\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-kube-api-access-jblqk\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:36.876392 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:36.876195    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563baac-7e8c-45f1-aa23-a6eb2430341d-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:37.101136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.101047    5551 generic.go:356] "Generic (PLEG): container finished" podID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerID="1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3" exitCode=0
Oct 31 18:53:37.101136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.101103    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"
Oct 31 18:53:37.101136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.101122    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" event={"ID":"1563baac-7e8c-45f1-aa23-a6eb2430341d","Type":"ContainerDied","Data":"1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3"}
Oct 31 18:53:37.101374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.101169    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll" event={"ID":"1563baac-7e8c-45f1-aa23-a6eb2430341d","Type":"ContainerDied","Data":"a624e747ca7069490d07af17e6b8a7cd90366ce9ab8f7eab8354fdb2ba607f68"}
Oct 31 18:53:37.101374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.101196    5551 scope.go:122] "RemoveContainer" containerID="1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3"
Oct 31 18:53:37.111569 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.111550    5551 scope.go:122] "RemoveContainer" containerID="59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf"
Oct 31 18:53:37.120621 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.120599    5551 scope.go:122] "RemoveContainer" containerID="1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3"
Oct 31 18:53:37.120884 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:37.120867    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3\": container with ID starting with 1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3 not found: ID does not exist" containerID="1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3"
Oct 31 18:53:37.120934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.120892    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3"} err="failed to get container status \"1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3\": rpc error: code = NotFound desc = could not find container \"1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3\": container with ID starting with 1141c158ff03894c4c9a5588ed96c3b3f895030e2475d44025a04218f9f054e3 not found: ID does not exist"
Oct 31 18:53:37.120934 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.120909    5551 scope.go:122] "RemoveContainer" containerID="59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf"
Oct 31 18:53:37.121125 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:37.121107    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf\": container with ID starting with 59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf not found: ID does not exist" containerID="59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf"
Oct 31 18:53:37.121166 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.121129    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf"} err="failed to get container status \"59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf\": rpc error: code = NotFound desc = could not find container \"59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf\": container with ID starting with 59f5d446b396e5e6a0f3029bd9a353c0e7c6dafa5e539fbd1c5af2b02bd69acf not found: ID does not exist"
Oct 31 18:53:37.265870 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.265837    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"]
Oct 31 18:53:37.270242 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:37.270214    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dc7b6-openshift-default-6944bcdfbf-dzcll"]
Oct 31 18:53:38.910324 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:38.910289    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" path="/var/lib/kubelet/pods/1563baac-7e8c-45f1-aa23-a6eb2430341d/volumes"
Oct 31 18:53:42.278209 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.278175    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"]
Oct 31 18:53:42.278986 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.278963    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4d23327e-95c2-432f-b8ba-cdb835d739ee" containerName="maas-api"
Oct 31 18:53:42.279096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.278989    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d23327e-95c2-432f-b8ba-cdb835d739ee" containerName="maas-api"
Oct 31 18:53:42.279096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279006    5551 cpu_manager.go:414] "containerMap: removing container" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy"
Oct 31 18:53:42.279096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279013    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy"
Oct 31 18:53:42.279096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279036    5551 cpu_manager.go:414] "containerMap: removing container" podUID="041cc576-f4f6-4acd-b95b-0c800cf0c6b3" containerName="payload-processing"
Oct 31 18:53:42.279096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279047    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041cc576-f4f6-4acd-b95b-0c800cf0c6b3" containerName="payload-processing"
Oct 31 18:53:42.279096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279071    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy"
Oct 31 18:53:42.279096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279077    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy"
Oct 31 18:53:42.279433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279109    5551 cpu_manager.go:414] "containerMap: removing container" podUID="ea8c9021-523b-4b02-a8eb-21cb2161156d" containerName="payload-pre-processing"
Oct 31 18:53:42.279433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279116    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8c9021-523b-4b02-a8eb-21cb2161156d" containerName="payload-pre-processing"
Oct 31 18:53:42.279433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279256    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d23327e-95c2-432f-b8ba-cdb835d739ee" containerName="maas-api"
Oct 31 18:53:42.279433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279273    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b4dc00-057e-4d84-a25c-31ed75e4a187" containerName="istio-proxy"
Oct 31 18:53:42.279433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279283    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="041cc576-f4f6-4acd-b95b-0c800cf0c6b3" containerName="payload-processing"
Oct 31 18:53:42.279433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279294    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy"
Oct 31 18:53:42.279433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279306    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8c9021-523b-4b02-a8eb-21cb2161156d" containerName="payload-pre-processing"
Oct 31 18:53:42.279433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.279316    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy"
Oct 31 18:53:42.284690 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.284665    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.336905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.336867    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e9c518-serving-cert\""
Oct 31 18:53:42.338673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.338639    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"]
Oct 31 18:53:42.360923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.360885    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50937ef-fa83-4104-837a-01a5f003023b-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.361083 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.361062    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-metrics-certs\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.361144 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.361099    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4f2\" (UniqueName: \"kubernetes.io/projected/b50937ef-fa83-4104-837a-01a5f003023b-kube-api-access-8p4f2\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.361217 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.361194    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.462548 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.462513    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50937ef-fa83-4104-837a-01a5f003023b-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.462751 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.462612    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-metrics-certs\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.462751 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.462640    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4f2\" (UniqueName: \"kubernetes.io/projected/b50937ef-fa83-4104-837a-01a5f003023b-kube-api-access-8p4f2\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.462751 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.462673    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.462936 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:42.462813    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-serving-cert: secret "maas-api-e2e-api-b-e9c518-serving-cert" not found
Oct 31 18:53:42.462936 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:42.462878    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls podName:b50937ef-fa83-4104-837a-01a5f003023b nodeName:}" failed. No retries permitted until 2026-10-31 18:53:42.962858828 +0000 UTC m=+1926.194097099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls") pod "maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf" (UID: "b50937ef-fa83-4104-837a-01a5f003023b") : secret "maas-api-e2e-api-b-e9c518-serving-cert" not found
Oct 31 18:53:42.463158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.463136    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50937ef-fa83-4104-837a-01a5f003023b-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.465156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.465134    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-metrics-certs\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.520565 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.520537    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4f2\" (UniqueName: \"kubernetes.io/projected/b50937ef-fa83-4104-837a-01a5f003023b-kube-api-access-8p4f2\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.969704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.969660    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:42.973228 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:42.973197    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls\") pod \"maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:43.201572 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:43.201543    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:43.582059 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:43.582022    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"]
Oct 31 18:53:43.585016 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:53:43.584990    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50937ef_fa83_4104_837a_01a5f003023b.slice/crio-e5e4e1f322c662ab5d49ae4f277f3708e544646280f5b502f134fa7c3b23e353 WatchSource:0}: Error finding container e5e4e1f322c662ab5d49ae4f277f3708e544646280f5b502f134fa7c3b23e353: Status 404 returned error can't find the container with id e5e4e1f322c662ab5d49ae4f277f3708e544646280f5b502f134fa7c3b23e353
Oct 31 18:53:43.586334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:43.586316    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:53:44.150448 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:44.150404    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf" event={"ID":"b50937ef-fa83-4104-837a-01a5f003023b","Type":"ContainerStarted","Data":"e5e4e1f322c662ab5d49ae4f277f3708e544646280f5b502f134fa7c3b23e353"}
Oct 31 18:53:45.160558 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:45.160517    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf" event={"ID":"b50937ef-fa83-4104-837a-01a5f003023b","Type":"ContainerStarted","Data":"252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d"}
Oct 31 18:53:45.160990 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:45.160623    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:45.276847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:45.276799    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf" podStartSLOduration=2.40962971 podStartE2EDuration="3.276785067s" totalImagesPullingTime="867.155357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:53:45.274046516 +0000 UTC m=+1928.505284803" watchObservedRunningTime="2026-10-31 18:53:45.276785067 +0000 UTC m=+1928.508023354"
Oct 31 18:53:46.608362 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:46.608294    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:53:50.874165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:50.874124    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"]
Oct 31 18:53:50.874698 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:50.874330    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy" containerID="cri-o://790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364" gracePeriod=30
Oct 31 18:53:51.173861 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:51.173835    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:53:52.928189 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:52.928156    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"]
Oct 31 18:53:52.928746 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:52.928732    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy"
Oct 31 18:53:52.928791 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:52.928747    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1563baac-7e8c-45f1-aa23-a6eb2430341d" containerName="istio-proxy"
Oct 31 18:53:52.940663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:52.940632    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:52.994314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:52.994283    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"]
Oct 31 18:53:53.019853 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.019817    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-socket\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.020005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.019917    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-data\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.020005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.019949    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-token\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.020005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.019970    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istiod-ca-cert\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.020141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.020086    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c824h\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-kube-api-access-c824h\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.020141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.020110    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-credential-socket\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.020141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.020129    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-certs\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.020268 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.020153    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-envoy\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.020306 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.020289    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-podinfo\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.039519 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.039488    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-84c46e-openshift-default-dockercfg-mbd6q\""
Oct 31 18:53:53.120863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.120834    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-podinfo\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121018 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.120883    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-socket\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121086 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121061    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-data\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121092    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-token\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121115    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istiod-ca-cert\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121190    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-socket\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121188    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c824h\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-kube-api-access-c824h\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121251    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-credential-socket\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121282    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-certs\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121324    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-envoy\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121394    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-data\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121591    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-credential-socket\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121613    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-certs\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.121895 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.121794    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istiod-ca-cert\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.123576 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.123555    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-podinfo\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.123670 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.123564    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-envoy\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.188357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.188240    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-token\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.200550 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.196139    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c824h\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-kube-api-access-c824h\") pod \"e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") " pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.255525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.255489    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:53.684819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.684789    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"]
Oct 31 18:53:53.685778 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:53:53.685748    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7999f6f9_a7e8_4ff7_a9de_57aefb668280.slice/crio-aa196a29b865aacc4a1db6b93429cf80fca62b11a0de36a14f8d00e781366b64 WatchSource:0}: Error finding container aa196a29b865aacc4a1db6b93429cf80fca62b11a0de36a14f8d00e781366b64: Status 404 returned error can't find the container with id aa196a29b865aacc4a1db6b93429cf80fca62b11a0de36a14f8d00e781366b64
Oct 31 18:53:53.688330 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.688297    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:53:53.688398 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.688362    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:53:53.688398 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:53.688391    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:53:54.227662 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:54.227625    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" event={"ID":"7999f6f9-a7e8-4ff7-a9de-57aefb668280","Type":"ContainerStarted","Data":"484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06"}
Oct 31 18:53:54.228128 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:54.227681    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" event={"ID":"7999f6f9-a7e8-4ff7-a9de-57aefb668280","Type":"ContainerStarted","Data":"aa196a29b865aacc4a1db6b93429cf80fca62b11a0de36a14f8d00e781366b64"}
Oct 31 18:53:54.255704 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:54.255679    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:54.261431 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:54.261408    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:54.347552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:54.347504    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" podStartSLOduration=2.34749038 podStartE2EDuration="2.34749038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:53:54.346631762 +0000 UTC m=+1937.577870053" watchObservedRunningTime="2026-10-31 18:53:54.34749038 +0000 UTC m=+1937.578728670"
Oct 31 18:53:55.234039 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:55.234008    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:55.235081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:55.235053    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:53:56.157715 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.157641    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:53:56.244229 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.244197    5551 generic.go:356] "Generic (PLEG): container finished" podID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerID="790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364" exitCode=0
Oct 31 18:53:56.244729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.244271    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"
Oct 31 18:53:56.244729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.244288    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" event={"ID":"604d091f-e6cc-4694-ae3f-783874d1e09e","Type":"ContainerDied","Data":"790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364"}
Oct 31 18:53:56.244729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.244338    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h" event={"ID":"604d091f-e6cc-4694-ae3f-783874d1e09e","Type":"ContainerDied","Data":"f5bc4df0fb12384fd2ada2c010a4f16af4f670a6f1e2625d5ba6bf91a4256471"}
Oct 31 18:53:56.244729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.244360    5551 scope.go:122] "RemoveContainer" containerID="790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364"
Oct 31 18:53:56.254528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.254510    5551 scope.go:122] "RemoveContainer" containerID="952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9"
Oct 31 18:53:56.258791 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.258770    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604d091f-e6cc-4694-ae3f-783874d1e09e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d091f-e6cc-4694-ae3f-783874d1e09e-istiod-ca-cert\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.258907 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.258835    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-socket\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.258907 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.258869    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-podinfo\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.259000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.258927    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-certs\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.259040 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259021    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-kube-api-access-h69lp\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-kube-api-access-h69lp\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.259074 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259039    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-envoy\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.259107 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259087    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-data\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.259141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259112    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-credential-socket\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.259174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259145    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-token\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-token\") pod \"604d091f-e6cc-4694-ae3f-783874d1e09e\" (UID: \"604d091f-e6cc-4694-ae3f-783874d1e09e\") "
Oct 31 18:53:56.259542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259520    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-certs" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:56.259764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259737    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-credential-socket" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:56.259910 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259674    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-socket" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:56.260027 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.260004    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604d091f-e6cc-4694-ae3f-783874d1e09e-istiod-ca-cert" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:56.260163 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.259596    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-data" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:56.262369 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.262339    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-envoy" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:56.262505 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.262481    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-kube-api-access-h69lp" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "kube-api-access-h69lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:56.262589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.262518    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-podinfo" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:53:56.264795 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.264781    5551 scope.go:122] "RemoveContainer" containerID="790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364"
Oct 31 18:53:56.265046 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:56.265026    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364\": container with ID starting with 790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364 not found: ID does not exist" containerID="790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364"
Oct 31 18:53:56.265091 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.265053    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364"} err="failed to get container status \"790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364\": rpc error: code = NotFound desc = could not find container \"790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364\": container with ID starting with 790842d12170e23745dd6d2daf577063e0d2a92776662181c4f9d065cbd74364 not found: ID does not exist"
Oct 31 18:53:56.265091 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.265069    5551 scope.go:122] "RemoveContainer" containerID="952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9"
Oct 31 18:53:56.265307 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:53:56.265294    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9\": container with ID starting with 952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9 not found: ID does not exist" containerID="952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9"
Oct 31 18:53:56.265360 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.265311    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9"} err="failed to get container status \"952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9\": rpc error: code = NotFound desc = could not find container \"952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9\": container with ID starting with 952ddff104cf57290a1bd532e96f34996a636cae502cbb2b2c18cba4157428f9 not found: ID does not exist"
Oct 31 18:53:56.269596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.269571    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-token" pod "604d091f-e6cc-4694-ae3f-783874d1e09e" (UID: "604d091f-e6cc-4694-ae3f-783874d1e09e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:56.361721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361672    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h69lp\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-kube-api-access-h69lp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.361721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361719    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.361933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361737    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.361933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361751    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.361933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361765    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.361933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361778    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d091f-e6cc-4694-ae3f-783874d1e09e-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.361933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361792    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.361933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361806    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d091f-e6cc-4694-ae3f-783874d1e09e-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.361933 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.361820    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d091f-e6cc-4694-ae3f-783874d1e09e-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:56.667863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.667795    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"]
Oct 31 18:53:56.675413 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.672908    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-100faf-openshift-default-644ff498c7-rcc6h"]
Oct 31 18:53:56.908813 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:53:56.908781    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" path="/var/lib/kubelet/pods/604d091f-e6cc-4694-ae3f-783874d1e09e/volumes"
Oct 31 18:54:16.604784 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:54:16.604746    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:54:39.072774 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:39.072746    5551 scope.go:122] "RemoveContainer" containerID="8897bb8ec178a8df99409821775ec5873dd3afcc7d5aaba7292df1f6c4aaec22"
Oct 31 18:54:46.605648 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:54:46.605550    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:54:50.994214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:50.994167    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"]
Oct 31 18:54:50.994831 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:50.994438    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc" podUID="d1178707-69aa-42df-9675-74cf68d4eef5" containerName="discovery" containerID="cri-o://35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e" gracePeriod=30
Oct 31 18:54:51.262720 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.262668    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:54:51.404736 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.404704    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-istio-token\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-istio-token\") pod \"d1178707-69aa-42df-9675-74cf68d4eef5\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") "
Oct 31 18:54:51.404898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.404801    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-kubeconfig\") pod \"d1178707-69aa-42df-9675-74cf68d4eef5\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") "
Oct 31 18:54:51.405011 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.404993    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1178707-69aa-42df-9675-74cf68d4eef5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1178707-69aa-42df-9675-74cf68d4eef5-local-certs\") pod \"d1178707-69aa-42df-9675-74cf68d4eef5\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") "
Oct 31 18:54:51.405053 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.405043    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-dns-cert\") pod \"d1178707-69aa-42df-9675-74cf68d4eef5\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") "
Oct 31 18:54:51.405117 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.405102    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-cacerts\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-cacerts\") pod \"d1178707-69aa-42df-9675-74cf68d4eef5\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") "
Oct 31 18:54:51.405190 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.405176    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-ca-configmap\") pod \"d1178707-69aa-42df-9675-74cf68d4eef5\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") "
Oct 31 18:54:51.405247 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.405199    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-kube-api-access-9zdhg\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-kube-api-access-9zdhg\") pod \"d1178707-69aa-42df-9675-74cf68d4eef5\" (UID: \"d1178707-69aa-42df-9675-74cf68d4eef5\") "
Oct 31 18:54:51.405943 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.405716    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-ca-configmap" pod "d1178707-69aa-42df-9675-74cf68d4eef5" (UID: "d1178707-69aa-42df-9675-74cf68d4eef5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:54:51.407323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.407295    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-kubeconfig" pod "d1178707-69aa-42df-9675-74cf68d4eef5" (UID: "d1178707-69aa-42df-9675-74cf68d4eef5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:54:51.407578 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.407547    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-istio-token" pod "d1178707-69aa-42df-9675-74cf68d4eef5" (UID: "d1178707-69aa-42df-9675-74cf68d4eef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:54:51.407925 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.407897    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1178707-69aa-42df-9675-74cf68d4eef5-local-certs" pod "d1178707-69aa-42df-9675-74cf68d4eef5" (UID: "d1178707-69aa-42df-9675-74cf68d4eef5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:54:51.408013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.407951    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-cacerts" pod "d1178707-69aa-42df-9675-74cf68d4eef5" (UID: "d1178707-69aa-42df-9675-74cf68d4eef5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:54:51.408013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.407995    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-kube-api-access-9zdhg" pod "d1178707-69aa-42df-9675-74cf68d4eef5" (UID: "d1178707-69aa-42df-9675-74cf68d4eef5"). InnerVolumeSpecName "kube-api-access-9zdhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:54:51.408090 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.408028    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-dns-cert" pod "d1178707-69aa-42df-9675-74cf68d4eef5" (UID: "d1178707-69aa-42df-9675-74cf68d4eef5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:54:51.507552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.507514    5551 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1178707-69aa-42df-9675-74cf68d4eef5-local-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:51.507552 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.507554    5551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-dns-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:51.507830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.507569    5551 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-cacerts\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:51.507830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.507584    5551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1178707-69aa-42df-9675-74cf68d4eef5-istio-csr-ca-configmap\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:51.507830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.507600    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zdhg\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-kube-api-access-9zdhg\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:51.507830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.507616    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1178707-69aa-42df-9675-74cf68d4eef5-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:51.507830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.507631    5551 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1178707-69aa-42df-9675-74cf68d4eef5-istio-kubeconfig\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:51.641221 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.641185    5551 generic.go:356] "Generic (PLEG): container finished" podID="d1178707-69aa-42df-9675-74cf68d4eef5" containerID="35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e" exitCode=0
Oct 31 18:54:51.641428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.641240    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"
Oct 31 18:54:51.641428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.641267    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc" event={"ID":"d1178707-69aa-42df-9675-74cf68d4eef5","Type":"ContainerDied","Data":"35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e"}
Oct 31 18:54:51.641428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.641321    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc" event={"ID":"d1178707-69aa-42df-9675-74cf68d4eef5","Type":"ContainerDied","Data":"e97b74637cbd903793c907d9183dcb63cfc84719c0886a9b8f2f3c6e810971b8"}
Oct 31 18:54:51.641428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.641342    5551 scope.go:122] "RemoveContainer" containerID="35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e"
Oct 31 18:54:51.654530 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.654511    5551 scope.go:122] "RemoveContainer" containerID="35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e"
Oct 31 18:54:51.654829 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:54:51.654811    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e\": container with ID starting with 35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e not found: ID does not exist" containerID="35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e"
Oct 31 18:54:51.654896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.654843    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e"} err="failed to get container status \"35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e\": rpc error: code = NotFound desc = could not find container \"35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e\": container with ID starting with 35394fe694deeba5851df8b4aaca3a06d08ec515001a38faa4061e2a6688498e not found: ID does not exist"
Oct 31 18:54:51.805761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.805725    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"]
Oct 31 18:54:51.810968 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:51.810942    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-897nc"]
Oct 31 18:54:52.909196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:52.909169    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1178707-69aa-42df-9675-74cf68d4eef5" path="/var/lib/kubelet/pods/d1178707-69aa-42df-9675-74cf68d4eef5/volumes"
Oct 31 18:54:56.526707 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.526665    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"]
Oct 31 18:54:56.527217 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.527204    5551 cpu_manager.go:414] "containerMap: removing container" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy"
Oct 31 18:54:56.527267 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.527219    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy"
Oct 31 18:54:56.527267 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.527244    5551 cpu_manager.go:414] "containerMap: removing container" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy"
Oct 31 18:54:56.527267 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.527250    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy"
Oct 31 18:54:56.527267 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.527257    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d1178707-69aa-42df-9675-74cf68d4eef5" containerName="discovery"
Oct 31 18:54:56.527267 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.527262    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1178707-69aa-42df-9675-74cf68d4eef5" containerName="discovery"
Oct 31 18:54:56.527419 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.527349    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy"
Oct 31 18:54:56.527419 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.527359    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1178707-69aa-42df-9675-74cf68d4eef5" containerName="discovery"
Oct 31 18:54:56.530619 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.530602    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562106    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-token\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562163    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-envoy\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562255    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-podinfo\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562303    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-data\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562329    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-certs\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562411    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istiod-ca-cert\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562477    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrbj8\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-kube-api-access-wrbj8\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562512    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-credential-socket\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.562642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.562596    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-socket\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.584769 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.584740    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-919b6e-openshift-default-dockercfg-8sn6v\""
Oct 31 18:54:56.636652 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.636619    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"]
Oct 31 18:54:56.665185 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.665152    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-socket\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.665553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.665529    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-token\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.665708 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.665692    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-envoy\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.665950 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.665928    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-podinfo\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.666111 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.666094    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-data\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.666234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.666218    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-certs\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.666353 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.666339    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istiod-ca-cert\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.666519 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.666500    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrbj8\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-kube-api-access-wrbj8\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.666655 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.666640    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-credential-socket\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.672339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.672303    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-data\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.672613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.672588    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-certs\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.673384 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.673347    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istiod-ca-cert\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.673786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.673763    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-credential-socket\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.674406 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.674382    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-socket\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.689696 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.686500    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-podinfo\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.698067 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.698002    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-envoy\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.731247 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.731212    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-token\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.731507 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.731453    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrbj8\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-kube-api-access-wrbj8\") pod \"e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") " pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:56.844540 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:56.844430    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:57.277929 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:57.277903    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"]
Oct 31 18:54:57.280023 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:54:57.279995    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bbd9f0_9646_4c5b_bf7f_a9b8a7b6e41d.slice/crio-83772de904c07b8d6b834888c5f1bf895785f4b61fe98a7215d50eca503f9a29 WatchSource:0}: Error finding container 83772de904c07b8d6b834888c5f1bf895785f4b61fe98a7215d50eca503f9a29: Status 404 returned error can't find the container with id 83772de904c07b8d6b834888c5f1bf895785f4b61fe98a7215d50eca503f9a29
Oct 31 18:54:57.282734 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:57.282702    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:54:57.282822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:57.282765    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:54:57.282822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:57.282795    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:54:57.709882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:57.709846    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" event={"ID":"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d","Type":"ContainerStarted","Data":"7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7"}
Oct 31 18:54:57.710263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:57.709900    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" event={"ID":"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d","Type":"ContainerStarted","Data":"83772de904c07b8d6b834888c5f1bf895785f4b61fe98a7215d50eca503f9a29"}
Oct 31 18:54:57.827010 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:57.826954    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podStartSLOduration=1.826939476 podStartE2EDuration="1.826939476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:57.823892398 +0000 UTC m=+2001.055130688" watchObservedRunningTime="2026-10-31 18:54:57.826939476 +0000 UTC m=+2001.058177767"
Oct 31 18:54:57.845001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:57.844971    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:58.851235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:58.851206    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:59.722494 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:59.722452    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:54:59.723426 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:54:59.723405    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:55:07.639734 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.639697    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"]
Oct 31 18:55:07.640347 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.640332    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="604d091f-e6cc-4694-ae3f-783874d1e09e" containerName="istio-proxy"
Oct 31 18:55:07.643810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.643790    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:07.700021 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.699989    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"]
Oct 31 18:55:07.752446 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.752415    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-919b6e\""
Oct 31 18:55:07.754874 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.752829    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-919b6e-dockercfg-vhf6z\""
Oct 31 18:55:07.807278 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.807241    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"]
Oct 31 18:55:07.812425 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.812400    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:07.815697 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.815639    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76mbm\" (UniqueName: \"kubernetes.io/projected/97a91f0c-6869-4205-ae00-8cfca222d370-kube-api-access-76mbm\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc\" (UID: \"97a91f0c-6869-4205-ae00-8cfca222d370\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:07.815848 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.815706    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a91f0c-6869-4205-ae00-8cfca222d370-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc\" (UID: \"97a91f0c-6869-4205-ae00-8cfca222d370\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:07.839140 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.839113    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"]
Oct 31 18:55:07.916493 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.916388    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85xl7\" (UniqueName: \"kubernetes.io/projected/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-kube-api-access-85xl7\") pod \"payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw\" (UID: \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:07.916634 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.916535    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76mbm\" (UniqueName: \"kubernetes.io/projected/97a91f0c-6869-4205-ae00-8cfca222d370-kube-api-access-76mbm\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc\" (UID: \"97a91f0c-6869-4205-ae00-8cfca222d370\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:07.916634 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.916584    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a91f0c-6869-4205-ae00-8cfca222d370-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc\" (UID: \"97a91f0c-6869-4205-ae00-8cfca222d370\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:07.916702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.916685    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw\" (UID: \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:07.917332 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.917314    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a91f0c-6869-4205-ae00-8cfca222d370-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc\" (UID: \"97a91f0c-6869-4205-ae00-8cfca222d370\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:07.973630 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:07.973595    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76mbm\" (UniqueName: \"kubernetes.io/projected/97a91f0c-6869-4205-ae00-8cfca222d370-kube-api-access-76mbm\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc\" (UID: \"97a91f0c-6869-4205-ae00-8cfca222d370\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:08.018136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.018110    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw\" (UID: \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:08.018289 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.018164    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85xl7\" (UniqueName: \"kubernetes.io/projected/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-kube-api-access-85xl7\") pod \"payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw\" (UID: \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:08.018825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.018803    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw\" (UID: \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:08.074658 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.074630    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85xl7\" (UniqueName: \"kubernetes.io/projected/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-kube-api-access-85xl7\") pod \"payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw\" (UID: \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:08.126742 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.126722    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:08.259539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.259510    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:08.503949 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.503919    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"]
Oct 31 18:55:08.505598 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:55:08.505571    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e1ae25_63b2_4ac0_95a1_9b7fcbbfc2e5.slice/crio-d58b297637d7bed77d02f7de6bb5b1c8c81b8af84685b228979f4b12ea2fe7ea WatchSource:0}: Error finding container d58b297637d7bed77d02f7de6bb5b1c8c81b8af84685b228979f4b12ea2fe7ea: Status 404 returned error can't find the container with id d58b297637d7bed77d02f7de6bb5b1c8c81b8af84685b228979f4b12ea2fe7ea
Oct 31 18:55:08.635902 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.635868    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"]
Oct 31 18:55:08.637851 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:55:08.637814    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a91f0c_6869_4205_ae00_8cfca222d370.slice/crio-6f0c92bd9c3e0a25a1f0021ce64f2a748e2ae33a8b97bce98524fe0824346258 WatchSource:0}: Error finding container 6f0c92bd9c3e0a25a1f0021ce64f2a748e2ae33a8b97bce98524fe0824346258: Status 404 returned error can't find the container with id 6f0c92bd9c3e0a25a1f0021ce64f2a748e2ae33a8b97bce98524fe0824346258
Oct 31 18:55:08.797214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.797076    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw" event={"ID":"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5","Type":"ContainerStarted","Data":"d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e"}
Oct 31 18:55:08.797214 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.797187    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw" event={"ID":"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5","Type":"ContainerStarted","Data":"d58b297637d7bed77d02f7de6bb5b1c8c81b8af84685b228979f4b12ea2fe7ea"}
Oct 31 18:55:08.797825 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.797227    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:08.799575 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.799533    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc" event={"ID":"97a91f0c-6869-4205-ae00-8cfca222d370","Type":"ContainerStarted","Data":"84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef"}
Oct 31 18:55:08.799775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.799607    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:08.799775 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.799629    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc" event={"ID":"97a91f0c-6869-4205-ae00-8cfca222d370","Type":"ContainerStarted","Data":"6f0c92bd9c3e0a25a1f0021ce64f2a748e2ae33a8b97bce98524fe0824346258"}
Oct 31 18:55:08.911423 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:08.911366    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw" podStartSLOduration=1.911351611 podStartE2EDuration="1.911351611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:55:08.90854798 +0000 UTC m=+2012.139786273" watchObservedRunningTime="2026-10-31 18:55:08.911351611 +0000 UTC m=+2012.142589933"
Oct 31 18:55:09.028589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:09.028506    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc" podStartSLOduration=2.023407938 podStartE2EDuration="2.023407938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:55:09.022843529 +0000 UTC m=+2012.254081819" watchObservedRunningTime="2026-10-31 18:55:09.023407938 +0000 UTC m=+2012.254646229"
Oct 31 18:55:11.394096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.394065    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"]
Oct 31 18:55:11.398207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.398189    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.451793 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.451750    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-84c46e-serving-cert\""
Oct 31 18:55:11.503781 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.503748    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"]
Oct 31 18:55:11.563125 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.563088    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.563323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.563283    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486e62d9-fead-4f93-aaa6-cc565db5aaf4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.563323 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.563310    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-metrics-certs\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.563547 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.563334    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlkh\" (UniqueName: \"kubernetes.io/projected/486e62d9-fead-4f93-aaa6-cc565db5aaf4-kube-api-access-mtlkh\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.664904 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.664814    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486e62d9-fead-4f93-aaa6-cc565db5aaf4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.664904 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.664854    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-metrics-certs\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.664904 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.664883    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlkh\" (UniqueName: \"kubernetes.io/projected/486e62d9-fead-4f93-aaa6-cc565db5aaf4-kube-api-access-mtlkh\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.665181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.665032    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.665689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.665656    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486e62d9-fead-4f93-aaa6-cc565db5aaf4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.667683 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.667650    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-metrics-certs\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.667786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.667690    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:11.723657 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:11.723630    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlkh\" (UniqueName: \"kubernetes.io/projected/486e62d9-fead-4f93-aaa6-cc565db5aaf4-kube-api-access-mtlkh\") pod \"maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:12.012886 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:12.012802    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:12.402883 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:12.402852    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"]
Oct 31 18:55:12.404862 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:55:12.404829    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486e62d9_fead_4f93_aaa6_cc565db5aaf4.slice/crio-2452a6f65fcec2a7dcb9234d45e5feaec2458ce6c8e4cb0df2f225b2bb86c1b4 WatchSource:0}: Error finding container 2452a6f65fcec2a7dcb9234d45e5feaec2458ce6c8e4cb0df2f225b2bb86c1b4: Status 404 returned error can't find the container with id 2452a6f65fcec2a7dcb9234d45e5feaec2458ce6c8e4cb0df2f225b2bb86c1b4
Oct 31 18:55:12.829855 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:12.829777    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k" event={"ID":"486e62d9-fead-4f93-aaa6-cc565db5aaf4","Type":"ContainerStarted","Data":"2452a6f65fcec2a7dcb9234d45e5feaec2458ce6c8e4cb0df2f225b2bb86c1b4"}
Oct 31 18:55:13.839965 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:13.839899    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k" event={"ID":"486e62d9-fead-4f93-aaa6-cc565db5aaf4","Type":"ContainerStarted","Data":"c98a04645bfe82327b7b55bfeb3b32b602151e7066c7813554c182a837c5e964"}
Oct 31 18:55:13.839965 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:13.839968    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:13.985489 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:13.985404    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k" podStartSLOduration=2.151116885 podStartE2EDuration="2.985385829s" totalImagesPullingTime="834.268944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:55:13.981900194 +0000 UTC m=+2017.213138494" watchObservedRunningTime="2026-10-31 18:55:13.985385829 +0000 UTC m=+2017.216624121"
Oct 31 18:55:16.605858 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:16.605821    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:55:19.809381 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:19.809344    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:55:19.809798 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:19.809419    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:55:19.889056 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:19.889027    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:55:22.875132 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:22.875101    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"]
Oct 31 18:55:22.875615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:22.875340    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh" podUID="d87eb73b-5383-423c-9496-54c6a1741afa" containerName="maas-api" containerID="cri-o://47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819" gracePeriod=30
Oct 31 18:55:23.182984 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.182953    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:55:23.235875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.234610    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87eb73b-5383-423c-9496-54c6a1741afa-kube-api-access-ppg9p\" (UniqueName: \"kubernetes.io/projected/d87eb73b-5383-423c-9496-54c6a1741afa-kube-api-access-ppg9p\") pod \"d87eb73b-5383-423c-9496-54c6a1741afa\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") "
Oct 31 18:55:23.235875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.234887    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87eb73b-5383-423c-9496-54c6a1741afa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87eb73b-5383-423c-9496-54c6a1741afa-openshift-service-ca\") pod \"d87eb73b-5383-423c-9496-54c6a1741afa\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") "
Oct 31 18:55:23.235875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.234947    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-metrics-certs\") pod \"d87eb73b-5383-423c-9496-54c6a1741afa\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") "
Oct 31 18:55:23.235875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.234994    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls\") pod \"d87eb73b-5383-423c-9496-54c6a1741afa\" (UID: \"d87eb73b-5383-423c-9496-54c6a1741afa\") "
Oct 31 18:55:23.235875 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.235641    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87eb73b-5383-423c-9496-54c6a1741afa-openshift-service-ca" pod "d87eb73b-5383-423c-9496-54c6a1741afa" (UID: "d87eb73b-5383-423c-9496-54c6a1741afa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:23.237300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.237270    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87eb73b-5383-423c-9496-54c6a1741afa-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:23.239618 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.238089    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87eb73b-5383-423c-9496-54c6a1741afa-kube-api-access-ppg9p" pod "d87eb73b-5383-423c-9496-54c6a1741afa" (UID: "d87eb73b-5383-423c-9496-54c6a1741afa"). InnerVolumeSpecName "kube-api-access-ppg9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:23.241201 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.241159    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls" pod "d87eb73b-5383-423c-9496-54c6a1741afa" (UID: "d87eb73b-5383-423c-9496-54c6a1741afa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:55:23.251778 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.245812    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-metrics-certs" pod "d87eb73b-5383-423c-9496-54c6a1741afa" (UID: "d87eb73b-5383-423c-9496-54c6a1741afa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:55:23.338325 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.338293    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:23.338325 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.338323    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87eb73b-5383-423c-9496-54c6a1741afa-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:23.338567 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.338335    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppg9p\" (UniqueName: \"kubernetes.io/projected/d87eb73b-5383-423c-9496-54c6a1741afa-kube-api-access-ppg9p\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:23.951949 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.951906    5551 generic.go:356] "Generic (PLEG): container finished" podID="d87eb73b-5383-423c-9496-54c6a1741afa" containerID="47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819" exitCode=0
Oct 31 18:55:23.952442 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.951963    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"
Oct 31 18:55:23.952442 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.951980    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh" event={"ID":"d87eb73b-5383-423c-9496-54c6a1741afa","Type":"ContainerDied","Data":"47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819"}
Oct 31 18:55:23.952442 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.952038    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh" event={"ID":"d87eb73b-5383-423c-9496-54c6a1741afa","Type":"ContainerDied","Data":"1fbf670f4f9fb84864be8ddd83cdb365de1302b66caa4c75f68352dd316cf3d6"}
Oct 31 18:55:23.952442 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.952063    5551 scope.go:122] "RemoveContainer" containerID="47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819"
Oct 31 18:55:23.963746 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.963732    5551 scope.go:122] "RemoveContainer" containerID="47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819"
Oct 31 18:55:23.964042 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:23.964021    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819\": container with ID starting with 47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819 not found: ID does not exist" containerID="47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819"
Oct 31 18:55:23.964098 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:23.964051    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819"} err="failed to get container status \"47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819\": rpc error: code = NotFound desc = could not find container \"47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819\": container with ID starting with 47a278d102f062d4bc2588643f240d270442fb094a9e7118609d4139259c2819 not found: ID does not exist"
Oct 31 18:55:24.114794 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:24.114768    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"]
Oct 31 18:55:24.161611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:24.161582    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2a4923-7c4b88bc84-gmblh"]
Oct 31 18:55:24.909529 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:24.909498    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87eb73b-5383-423c-9496-54c6a1741afa" path="/var/lib/kubelet/pods/d87eb73b-5383-423c-9496-54c6a1741afa/volumes"
Oct 31 18:55:28.403691 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:28.403660    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"]
Oct 31 18:55:28.404118 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:28.403886    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy" containerID="cri-o://7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c" gracePeriod=30
Oct 31 18:55:33.697027 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.696991    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:55:33.804869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.804834    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-credential-socket\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.804869 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.804875    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-envoy\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.805106 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.804937    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-certs\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.805106 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805006    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-socket\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.805106 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805046    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-data\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.805270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805155    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istiod-ca-cert\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.805270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805148    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-credential-socket" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:33.805386 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805246    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-certs" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:33.805386 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805311    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-kube-api-access-26hkj\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-kube-api-access-26hkj\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.805386 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805313    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-socket" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:33.805386 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805356    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-data" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:33.805386 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805377    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-token\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-token\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.805816 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805408    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-podinfo\") pod \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\" (UID: \"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705\") "
Oct 31 18:55:33.805816 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805661    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istiod-ca-cert" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:33.806208 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805965    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:33.806208 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.805991    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:33.806208 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.806001    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:33.806208 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.806010    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:33.806208 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.806023    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:33.808383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.808346    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-podinfo" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:55:33.808383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.808362    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-token" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:33.808571 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.808431    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-kube-api-access-26hkj" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "kube-api-access-26hkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:33.808571 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.808437    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-envoy" pod "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" (UID: "1e02db3a-b339-4d5c-b8f0-b1ea23f6b705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:33.907612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.907576    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hkj\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-kube-api-access-26hkj\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:33.907612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.907614    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:33.907798 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.907629    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:33.907798 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:33.907642    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:34.027820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.027739    5551 generic.go:356] "Generic (PLEG): container finished" podID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerID="7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c" exitCode=0
Oct 31 18:55:34.027820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.027787    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"
Oct 31 18:55:34.027983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.027829    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" event={"ID":"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705","Type":"ContainerDied","Data":"7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c"}
Oct 31 18:55:34.027983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.027886    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf" event={"ID":"1e02db3a-b339-4d5c-b8f0-b1ea23f6b705","Type":"ContainerDied","Data":"20f05af8caa4c83880e7e02b3f1c00a812ace7a064be76ebbb99cc07d937bdf5"}
Oct 31 18:55:34.027983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.027914    5551 scope.go:122] "RemoveContainer" containerID="7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c"
Oct 31 18:55:34.040007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.039989    5551 scope.go:122] "RemoveContainer" containerID="6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476"
Oct 31 18:55:34.048952 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.048936    5551 scope.go:122] "RemoveContainer" containerID="7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c"
Oct 31 18:55:34.049217 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:34.049201    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c\": container with ID starting with 7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c not found: ID does not exist" containerID="7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c"
Oct 31 18:55:34.049263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.049232    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c"} err="failed to get container status \"7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c\": rpc error: code = NotFound desc = could not find container \"7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c\": container with ID starting with 7765d39bfbd66951cee10f6ea22db8f4a895059067fa64c0dab45efb17801f9c not found: ID does not exist"
Oct 31 18:55:34.049263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.049248    5551 scope.go:122] "RemoveContainer" containerID="6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476"
Oct 31 18:55:34.049609 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:34.049583    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476\": container with ID starting with 6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476 not found: ID does not exist" containerID="6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476"
Oct 31 18:55:34.049721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.049613    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476"} err="failed to get container status \"6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476\": rpc error: code = NotFound desc = could not find container \"6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476\": container with ID starting with 6991a0a6f2c4b180eb24c8091bb852f0ff6c0c1bf2942fb275312522c872f476 not found: ID does not exist"
Oct 31 18:55:34.192780 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.192724    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"]
Oct 31 18:55:34.195761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.195733    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a4923-openshift-default-84cfcb4bd5-mj6rf"]
Oct 31 18:55:34.909332 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:34.909299    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" path="/var/lib/kubelet/pods/1e02db3a-b339-4d5c-b8f0-b1ea23f6b705/volumes"
Oct 31 18:55:41.140395 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.140359    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"]
Oct 31 18:55:41.140902 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.140632    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78" podUID="46f8622a-90a4-4c6b-898c-dcb95241545e" containerName="payload-processing" containerID="cri-o://a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95" gracePeriod=30
Oct 31 18:55:41.192054 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.192012    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"]
Oct 31 18:55:41.192270 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.192228    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr" podUID="66279707-0137-4a71-87fb-da0938885850" containerName="payload-pre-processing" containerID="cri-o://1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0" gracePeriod=30
Oct 31 18:55:41.498521 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.498492    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:55:41.501845 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.501823    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:55:41.604739 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.604692    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66279707-0137-4a71-87fb-da0938885850-kube-api-access-rtfbp\" (UniqueName: \"kubernetes.io/projected/66279707-0137-4a71-87fb-da0938885850-kube-api-access-rtfbp\") pod \"66279707-0137-4a71-87fb-da0938885850\" (UID: \"66279707-0137-4a71-87fb-da0938885850\") "
Oct 31 18:55:41.604739 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.604750    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66279707-0137-4a71-87fb-da0938885850-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66279707-0137-4a71-87fb-da0938885850-plugins-config-volume\") pod \"66279707-0137-4a71-87fb-da0938885850\" (UID: \"66279707-0137-4a71-87fb-da0938885850\") "
Oct 31 18:55:41.604975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.604909    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f8622a-90a4-4c6b-898c-dcb95241545e-kube-api-access-phlfq\" (UniqueName: \"kubernetes.io/projected/46f8622a-90a4-4c6b-898c-dcb95241545e-kube-api-access-phlfq\") pod \"46f8622a-90a4-4c6b-898c-dcb95241545e\" (UID: \"46f8622a-90a4-4c6b-898c-dcb95241545e\") "
Oct 31 18:55:41.604975 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.604952    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f8622a-90a4-4c6b-898c-dcb95241545e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f8622a-90a4-4c6b-898c-dcb95241545e-plugins-config-volume\") pod \"46f8622a-90a4-4c6b-898c-dcb95241545e\" (UID: \"46f8622a-90a4-4c6b-898c-dcb95241545e\") "
Oct 31 18:55:41.605194 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.605168    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66279707-0137-4a71-87fb-da0938885850-plugins-config-volume" pod "66279707-0137-4a71-87fb-da0938885850" (UID: "66279707-0137-4a71-87fb-da0938885850"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:41.605402 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.605381    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f8622a-90a4-4c6b-898c-dcb95241545e-plugins-config-volume" pod "46f8622a-90a4-4c6b-898c-dcb95241545e" (UID: "46f8622a-90a4-4c6b-898c-dcb95241545e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:41.605678 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.605575    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f8622a-90a4-4c6b-898c-dcb95241545e-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:41.605732 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.605724    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66279707-0137-4a71-87fb-da0938885850-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:41.607176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.607148    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f8622a-90a4-4c6b-898c-dcb95241545e-kube-api-access-phlfq" pod "46f8622a-90a4-4c6b-898c-dcb95241545e" (UID: "46f8622a-90a4-4c6b-898c-dcb95241545e"). InnerVolumeSpecName "kube-api-access-phlfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:41.607277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.607173    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66279707-0137-4a71-87fb-da0938885850-kube-api-access-rtfbp" pod "66279707-0137-4a71-87fb-da0938885850" (UID: "66279707-0137-4a71-87fb-da0938885850"). InnerVolumeSpecName "kube-api-access-rtfbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:41.718576 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.710403    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phlfq\" (UniqueName: \"kubernetes.io/projected/46f8622a-90a4-4c6b-898c-dcb95241545e-kube-api-access-phlfq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:41.718810 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:41.718785    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtfbp\" (UniqueName: \"kubernetes.io/projected/66279707-0137-4a71-87fb-da0938885850-kube-api-access-rtfbp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:42.092109 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.092014    5551 generic.go:356] "Generic (PLEG): container finished" podID="46f8622a-90a4-4c6b-898c-dcb95241545e" containerID="a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95" exitCode=0
Oct 31 18:55:42.092109 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.092067    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"
Oct 31 18:55:42.092360 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.092064    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78" event={"ID":"46f8622a-90a4-4c6b-898c-dcb95241545e","Type":"ContainerDied","Data":"a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95"}
Oct 31 18:55:42.092360 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.092185    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78" event={"ID":"46f8622a-90a4-4c6b-898c-dcb95241545e","Type":"ContainerDied","Data":"ec6e1beb6bef612c696cabbaff08814625ba701fd3f8c7a0ab3d364afa79fcb5"}
Oct 31 18:55:42.092360 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.092229    5551 scope.go:122] "RemoveContainer" containerID="a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95"
Oct 31 18:55:42.093623 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.093601    5551 generic.go:356] "Generic (PLEG): container finished" podID="66279707-0137-4a71-87fb-da0938885850" containerID="1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0" exitCode=0
Oct 31 18:55:42.093735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.093657    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"
Oct 31 18:55:42.093735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.093674    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr" event={"ID":"66279707-0137-4a71-87fb-da0938885850","Type":"ContainerDied","Data":"1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0"}
Oct 31 18:55:42.093735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.093713    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr" event={"ID":"66279707-0137-4a71-87fb-da0938885850","Type":"ContainerDied","Data":"2478bd7dd9c2212dae32a6fb1b2700b03a4bb6b464e79894ecc418d89f533efe"}
Oct 31 18:55:42.104196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.104160    5551 scope.go:122] "RemoveContainer" containerID="a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95"
Oct 31 18:55:42.104487 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:42.104451    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95\": container with ID starting with a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95 not found: ID does not exist" containerID="a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95"
Oct 31 18:55:42.104566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.104492    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95"} err="failed to get container status \"a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95\": rpc error: code = NotFound desc = could not find container \"a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95\": container with ID starting with a6a0939d9afdd201fb4a68b3dbeda4765e1fe6fb80616e734c480bc1cec5dc95 not found: ID does not exist"
Oct 31 18:55:42.104566 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.104517    5551 scope.go:122] "RemoveContainer" containerID="1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0"
Oct 31 18:55:42.113946 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.113932    5551 scope.go:122] "RemoveContainer" containerID="1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0"
Oct 31 18:55:42.114183 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:42.114158    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0\": container with ID starting with 1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0 not found: ID does not exist" containerID="1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0"
Oct 31 18:55:42.114243 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.114192    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0"} err="failed to get container status \"1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0\": rpc error: code = NotFound desc = could not find container \"1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0\": container with ID starting with 1c898a989750854df3f8d84aff1486ca1f14e1221603eac6ee19a2ff2b207cd0 not found: ID does not exist"
Oct 31 18:55:42.255737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.255711    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"]
Oct 31 18:55:42.258802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.258777    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-55ccfc5c5-2xjwr"]
Oct 31 18:55:42.365662 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.365576    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"]
Oct 31 18:55:42.368226 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.368195    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-68c4fc6d4-8ln78"]
Oct 31 18:55:42.909710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.909676    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f8622a-90a4-4c6b-898c-dcb95241545e" path="/var/lib/kubelet/pods/46f8622a-90a4-4c6b-898c-dcb95241545e/volumes"
Oct 31 18:55:42.910317 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:42.910293    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66279707-0137-4a71-87fb-da0938885850" path="/var/lib/kubelet/pods/66279707-0137-4a71-87fb-da0938885850/volumes"
Oct 31 18:55:46.605285 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:46.605242    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:55:48.849137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:48.849101    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"]
Oct 31 18:55:48.849612 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:48.849326    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk" podUID="02eebe21-8f57-4df3-a107-122eb0d46678" containerName="maas-api" containerID="cri-o://bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187" gracePeriod=30
Oct 31 18:55:49.139859 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.139833    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:55:49.144265 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.144239    5551 generic.go:356] "Generic (PLEG): container finished" podID="02eebe21-8f57-4df3-a107-122eb0d46678" containerID="bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187" exitCode=0
Oct 31 18:55:49.144363 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.144284    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"
Oct 31 18:55:49.144363 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.144317    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk" event={"ID":"02eebe21-8f57-4df3-a107-122eb0d46678","Type":"ContainerDied","Data":"bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187"}
Oct 31 18:55:49.144466 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.144367    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk" event={"ID":"02eebe21-8f57-4df3-a107-122eb0d46678","Type":"ContainerDied","Data":"cf0355aaad65954bd1c050db9591bdaf3b98f5c320bac5e75ee0cca0b9850f3b"}
Oct 31 18:55:49.144466 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.144388    5551 scope.go:122] "RemoveContainer" containerID="bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187"
Oct 31 18:55:49.160659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.160640    5551 scope.go:122] "RemoveContainer" containerID="bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187"
Oct 31 18:55:49.161063 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:49.161042    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187\": container with ID starting with bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187 not found: ID does not exist" containerID="bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187"
Oct 31 18:55:49.161146 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.161073    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187"} err="failed to get container status \"bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187\": rpc error: code = NotFound desc = could not find container \"bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187\": container with ID starting with bb7abb3a8d5128e292cfa8d993eb3d960c5356fef9c8745e9d8f22fe5e2a7187 not found: ID does not exist"
Oct 31 18:55:49.307606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.307573    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02eebe21-8f57-4df3-a107-122eb0d46678-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eebe21-8f57-4df3-a107-122eb0d46678-openshift-service-ca\") pod \"02eebe21-8f57-4df3-a107-122eb0d46678\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") "
Oct 31 18:55:49.307782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.307703    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-metrics-certs\") pod \"02eebe21-8f57-4df3-a107-122eb0d46678\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") "
Oct 31 18:55:49.307782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.307725    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls\") pod \"02eebe21-8f57-4df3-a107-122eb0d46678\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") "
Oct 31 18:55:49.307782 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.307751    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eebe21-8f57-4df3-a107-122eb0d46678-kube-api-access-dqrbb\" (UniqueName: \"kubernetes.io/projected/02eebe21-8f57-4df3-a107-122eb0d46678-kube-api-access-dqrbb\") pod \"02eebe21-8f57-4df3-a107-122eb0d46678\" (UID: \"02eebe21-8f57-4df3-a107-122eb0d46678\") "
Oct 31 18:55:49.307991 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.307961    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eebe21-8f57-4df3-a107-122eb0d46678-openshift-service-ca" pod "02eebe21-8f57-4df3-a107-122eb0d46678" (UID: "02eebe21-8f57-4df3-a107-122eb0d46678"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:49.308189 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.308169    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eebe21-8f57-4df3-a107-122eb0d46678-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:49.310102 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.310072    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls" pod "02eebe21-8f57-4df3-a107-122eb0d46678" (UID: "02eebe21-8f57-4df3-a107-122eb0d46678"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:55:49.310191 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.310152    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eebe21-8f57-4df3-a107-122eb0d46678-kube-api-access-dqrbb" pod "02eebe21-8f57-4df3-a107-122eb0d46678" (UID: "02eebe21-8f57-4df3-a107-122eb0d46678"). InnerVolumeSpecName "kube-api-access-dqrbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:49.310241 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.310198    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-metrics-certs" pod "02eebe21-8f57-4df3-a107-122eb0d46678" (UID: "02eebe21-8f57-4df3-a107-122eb0d46678"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:55:49.409400 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.409363    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:49.409400 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.409398    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02eebe21-8f57-4df3-a107-122eb0d46678-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:49.409676 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.409414    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrbb\" (UniqueName: \"kubernetes.io/projected/02eebe21-8f57-4df3-a107-122eb0d46678-kube-api-access-dqrbb\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:49.422386 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.422351    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"]
Oct 31 18:55:49.422923 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.422651    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy" containerID="cri-o://79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158" gracePeriod=30
Oct 31 18:55:49.684132 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.684048    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"]
Oct 31 18:55:49.688216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:49.688188    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aa4693-66cd556984-qnlvk"]
Oct 31 18:55:50.115165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:50.115083    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:55:50.115165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:50.115144    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:55:50.909312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:50.909282    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02eebe21-8f57-4df3-a107-122eb0d46678" path="/var/lib/kubelet/pods/02eebe21-8f57-4df3-a107-122eb0d46678/volumes"
Oct 31 18:55:52.581289 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:52.581250    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"]
Oct 31 18:55:52.581800 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:52.581553    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy" containerID="cri-o://38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582" gracePeriod=30
Oct 31 18:55:54.714930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.714897    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:55:54.885912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.885881    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-kube-api-access-f8g5r\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-kube-api-access-f8g5r\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.886091 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.885947    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-podinfo\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.886091 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886012    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-token\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-token\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.886091 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886041    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6038f7c4-9cd7-445a-bde8-952014f798fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6038f7c4-9cd7-445a-bde8-952014f798fb-istiod-ca-cert\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.886275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886143    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-socket\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.886275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886180    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-data\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.886275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886206    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-envoy\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.886275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886257    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-certs\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.886520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886337    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-credential-socket\") pod \"6038f7c4-9cd7-445a-bde8-952014f798fb\" (UID: \"6038f7c4-9cd7-445a-bde8-952014f798fb\") "
Oct 31 18:55:54.887597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886777    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-data" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:54.887597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886947    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-socket" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:54.887597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.886980    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-credential-socket" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:54.887597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.887360    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6038f7c4-9cd7-445a-bde8-952014f798fb-istiod-ca-cert" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:54.887597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.887565    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-certs" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:54.888717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.888681    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-kube-api-access-f8g5r" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "kube-api-access-f8g5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:54.888836 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.888814    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-podinfo" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:55:54.888967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.888943    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-token" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:54.889507 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.889482    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-envoy" pod "6038f7c4-9cd7-445a-bde8-952014f798fb" (UID: "6038f7c4-9cd7-445a-bde8-952014f798fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:54.987890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987807    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:54.987890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987838    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:54.987890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987853    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:54.987890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987868    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:54.987890 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987882    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6038f7c4-9cd7-445a-bde8-952014f798fb-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:54.988278 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987896    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8g5r\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-kube-api-access-f8g5r\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:54.988278 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987909    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:54.988278 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987923    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6038f7c4-9cd7-445a-bde8-952014f798fb-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:54.988278 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:54.987946    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6038f7c4-9cd7-445a-bde8-952014f798fb-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:55.187198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.187160    5551 generic.go:356] "Generic (PLEG): container finished" podID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerID="79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158" exitCode=0
Oct 31 18:55:55.187357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.187220    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"
Oct 31 18:55:55.187357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.187238    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" event={"ID":"6038f7c4-9cd7-445a-bde8-952014f798fb","Type":"ContainerDied","Data":"79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158"}
Oct 31 18:55:55.187357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.187296    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b" event={"ID":"6038f7c4-9cd7-445a-bde8-952014f798fb","Type":"ContainerDied","Data":"8857e9dafd46b8e23aaadb039297f3971e31ec1ed0ea3d94275f933d926e7fdd"}
Oct 31 18:55:55.187357 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.187321    5551 scope.go:122] "RemoveContainer" containerID="79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158"
Oct 31 18:55:55.197315 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.197297    5551 scope.go:122] "RemoveContainer" containerID="1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2"
Oct 31 18:55:55.206429 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.206413    5551 scope.go:122] "RemoveContainer" containerID="79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158"
Oct 31 18:55:55.206701 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:55.206685    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158\": container with ID starting with 79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158 not found: ID does not exist" containerID="79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158"
Oct 31 18:55:55.206770 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.206712    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158"} err="failed to get container status \"79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158\": rpc error: code = NotFound desc = could not find container \"79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158\": container with ID starting with 79675991ff3fe23561a86f0e5d1b5c040a01137bc681532d90038014b727a158 not found: ID does not exist"
Oct 31 18:55:55.206770 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.206728    5551 scope.go:122] "RemoveContainer" containerID="1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2"
Oct 31 18:55:55.206960 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:55.206942    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2\": container with ID starting with 1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2 not found: ID does not exist" containerID="1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2"
Oct 31 18:55:55.207004 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.206963    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2"} err="failed to get container status \"1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2\": rpc error: code = NotFound desc = could not find container \"1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2\": container with ID starting with 1e2adb3428dcd488d68954bee6f670809afb06fe37cf18b31a650a4551a59eb2 not found: ID does not exist"
Oct 31 18:55:55.353501 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.353375    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"]
Oct 31 18:55:55.403248 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:55.403211    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5d1b3f-openshift-default-7d56fbbb77-fww7b"]
Oct 31 18:55:56.911411 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:56.911375    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" path="/var/lib/kubelet/pods/6038f7c4-9cd7-445a-bde8-952014f798fb/volumes"
Oct 31 18:55:57.850187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:57.850130    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:55:58.024741 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.024711    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-credential-socket\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.024768    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-kube-api-access-9d2p7\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-kube-api-access-9d2p7\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.024787    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-podinfo\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.024833    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040453dc-9c8e-4741-a823-398aa8e3bc61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040453dc-9c8e-4741-a823-398aa8e3bc61-istiod-ca-cert\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.024861    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-envoy\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.024959    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-data\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025031    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-certs\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025030    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-credential-socket" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025098    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-socket\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025199 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025125    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-token\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-token\") pod \"040453dc-9c8e-4741-a823-398aa8e3bc61\" (UID: \"040453dc-9c8e-4741-a823-398aa8e3bc61\") "
Oct 31 18:55:58.025737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025255    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-data" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:58.025737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025500    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-certs" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:58.025737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025560    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040453dc-9c8e-4741-a823-398aa8e3bc61-istiod-ca-cert" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:58.025737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025712    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040453dc-9c8e-4741-a823-398aa8e3bc61-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.025737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025718    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-socket" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:58.025737 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025731    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.026042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025743    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.026042 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.025757    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.028200 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.028174    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-kube-api-access-9d2p7" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "kube-api-access-9d2p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:58.028342 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.028209    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-token" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:58.028342 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.028321    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-podinfo" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:55:58.028605 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.028561    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-envoy" pod "040453dc-9c8e-4741-a823-398aa8e3bc61" (UID: "040453dc-9c8e-4741-a823-398aa8e3bc61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:55:58.127405 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.127376    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.127405 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.127404    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.127605 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.127414    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2p7\" (UniqueName: \"kubernetes.io/projected/040453dc-9c8e-4741-a823-398aa8e3bc61-kube-api-access-9d2p7\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.127605 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.127423    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.127605 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.127432    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040453dc-9c8e-4741-a823-398aa8e3bc61-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:58.217868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.217835    5551 generic.go:356] "Generic (PLEG): container finished" podID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerID="38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582" exitCode=0
Oct 31 18:55:58.218068 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.217906    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"
Oct 31 18:55:58.218068 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.217960    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" event={"ID":"040453dc-9c8e-4741-a823-398aa8e3bc61","Type":"ContainerDied","Data":"38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582"}
Oct 31 18:55:58.218068 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.218023    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759" event={"ID":"040453dc-9c8e-4741-a823-398aa8e3bc61","Type":"ContainerDied","Data":"67d3374c227c157209d1f17a241623a393ccc32d628b3d7f6325832a3cff3f97"}
Oct 31 18:55:58.218068 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.218047    5551 scope.go:122] "RemoveContainer" containerID="38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582"
Oct 31 18:55:58.229088 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.229070    5551 scope.go:122] "RemoveContainer" containerID="208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad"
Oct 31 18:55:58.238450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.238431    5551 scope.go:122] "RemoveContainer" containerID="38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582"
Oct 31 18:55:58.238799 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:58.238765    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582\": container with ID starting with 38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582 not found: ID does not exist" containerID="38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582"
Oct 31 18:55:58.238883 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.238809    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582"} err="failed to get container status \"38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582\": rpc error: code = NotFound desc = could not find container \"38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582\": container with ID starting with 38fa80c2d050405084fd58de15ebb49f946e55796a3ee7f4d28e82b05d040582 not found: ID does not exist"
Oct 31 18:55:58.238883 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.238832    5551 scope.go:122] "RemoveContainer" containerID="208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad"
Oct 31 18:55:58.239131 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:55:58.239114    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad\": container with ID starting with 208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad not found: ID does not exist" containerID="208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad"
Oct 31 18:55:58.239173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.239138    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad"} err="failed to get container status \"208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad\": rpc error: code = NotFound desc = could not find container \"208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad\": container with ID starting with 208a3ec05ad607fa12595e42fcb78a9828b38951ba297c33719b103e16fbfaad not found: ID does not exist"
Oct 31 18:55:58.384685 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.384585    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"]
Oct 31 18:55:58.432473 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.432425    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-aa4693-openshift-default-5c69d87457-b4759"]
Oct 31 18:55:58.909994 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:55:58.909962    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" path="/var/lib/kubelet/pods/040453dc-9c8e-4741-a823-398aa8e3bc61/volumes"
Oct 31 18:56:00.571615 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:00.571542    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"]
Oct 31 18:56:00.572749 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:00.572716    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf" podUID="b50937ef-fa83-4104-837a-01a5f003023b" containerName="maas-api" containerID="cri-o://252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d" gracePeriod=30
Oct 31 18:56:00.883272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:00.883244    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:56:01.079378 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.079346    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls\") pod \"b50937ef-fa83-4104-837a-01a5f003023b\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") "
Oct 31 18:56:01.079570 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.079440    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50937ef-fa83-4104-837a-01a5f003023b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50937ef-fa83-4104-837a-01a5f003023b-openshift-service-ca\") pod \"b50937ef-fa83-4104-837a-01a5f003023b\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") "
Oct 31 18:56:01.079665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.079649    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-metrics-certs\") pod \"b50937ef-fa83-4104-837a-01a5f003023b\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") "
Oct 31 18:56:01.079722 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.079696    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50937ef-fa83-4104-837a-01a5f003023b-kube-api-access-8p4f2\" (UniqueName: \"kubernetes.io/projected/b50937ef-fa83-4104-837a-01a5f003023b-kube-api-access-8p4f2\") pod \"b50937ef-fa83-4104-837a-01a5f003023b\" (UID: \"b50937ef-fa83-4104-837a-01a5f003023b\") "
Oct 31 18:56:01.079850 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.079822    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50937ef-fa83-4104-837a-01a5f003023b-openshift-service-ca" pod "b50937ef-fa83-4104-837a-01a5f003023b" (UID: "b50937ef-fa83-4104-837a-01a5f003023b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:01.080177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.080155    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50937ef-fa83-4104-837a-01a5f003023b-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:01.082035 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.082007    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-metrics-certs" pod "b50937ef-fa83-4104-837a-01a5f003023b" (UID: "b50937ef-fa83-4104-837a-01a5f003023b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:56:01.082378 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.082357    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50937ef-fa83-4104-837a-01a5f003023b-kube-api-access-8p4f2" pod "b50937ef-fa83-4104-837a-01a5f003023b" (UID: "b50937ef-fa83-4104-837a-01a5f003023b"). InnerVolumeSpecName "kube-api-access-8p4f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:01.083761 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.083741    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls" pod "b50937ef-fa83-4104-837a-01a5f003023b" (UID: "b50937ef-fa83-4104-837a-01a5f003023b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:56:01.181097 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.181022    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:01.181097 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.181059    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4f2\" (UniqueName: \"kubernetes.io/projected/b50937ef-fa83-4104-837a-01a5f003023b-kube-api-access-8p4f2\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:01.181097 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.181072    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b50937ef-fa83-4104-837a-01a5f003023b-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:01.253115 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.253061    5551 generic.go:356] "Generic (PLEG): container finished" podID="b50937ef-fa83-4104-837a-01a5f003023b" containerID="252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d" exitCode=0
Oct 31 18:56:01.253298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.253232    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"
Oct 31 18:56:01.253298 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.253271    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf" event={"ID":"b50937ef-fa83-4104-837a-01a5f003023b","Type":"ContainerDied","Data":"252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d"}
Oct 31 18:56:01.253492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.253341    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf" event={"ID":"b50937ef-fa83-4104-837a-01a5f003023b","Type":"ContainerDied","Data":"e5e4e1f322c662ab5d49ae4f277f3708e544646280f5b502f134fa7c3b23e353"}
Oct 31 18:56:01.253492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.253376    5551 scope.go:122] "RemoveContainer" containerID="252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d"
Oct 31 18:56:01.267729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.267691    5551 scope.go:122] "RemoveContainer" containerID="252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d"
Oct 31 18:56:01.268024 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:56:01.268001    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d\": container with ID starting with 252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d not found: ID does not exist" containerID="252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d"
Oct 31 18:56:01.268090 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.268034    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d"} err="failed to get container status \"252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d\": rpc error: code = NotFound desc = could not find container \"252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d\": container with ID starting with 252562f508d26cf19d778a58cc4ec0e62f4207806edac619334df5bae61b466d not found: ID does not exist"
Oct 31 18:56:01.417334 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.417294    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"]
Oct 31 18:56:01.467684 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:01.467606    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9c518-598cff5df6-pwwqf"]
Oct 31 18:56:02.183339 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.183307    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"]
Oct 31 18:56:02.184025 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184006    5551 cpu_manager.go:414] "containerMap: removing container" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy"
Oct 31 18:56:02.184072 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184028    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy"
Oct 31 18:56:02.184072 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184042    5551 cpu_manager.go:414] "containerMap: removing container" podUID="02eebe21-8f57-4df3-a107-122eb0d46678" containerName="maas-api"
Oct 31 18:56:02.184072 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184047    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eebe21-8f57-4df3-a107-122eb0d46678" containerName="maas-api"
Oct 31 18:56:02.184072 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184055    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy"
Oct 31 18:56:02.184072 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184060    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184085    5551 cpu_manager.go:414] "containerMap: removing container" podUID="46f8622a-90a4-4c6b-898c-dcb95241545e" containerName="payload-processing"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184090    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f8622a-90a4-4c6b-898c-dcb95241545e" containerName="payload-processing"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184096    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b50937ef-fa83-4104-837a-01a5f003023b" containerName="maas-api"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184100    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50937ef-fa83-4104-837a-01a5f003023b" containerName="maas-api"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184107    5551 cpu_manager.go:414] "containerMap: removing container" podUID="66279707-0137-4a71-87fb-da0938885850" containerName="payload-pre-processing"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184112    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66279707-0137-4a71-87fb-da0938885850" containerName="payload-pre-processing"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184132    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184137    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184145    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d87eb73b-5383-423c-9496-54c6a1741afa" containerName="maas-api"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184149    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87eb73b-5383-423c-9496-54c6a1741afa" containerName="maas-api"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184159    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy"
Oct 31 18:56:02.184225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184164    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184238    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184248    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184253    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184261    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87eb73b-5383-423c-9496-54c6a1741afa" containerName="maas-api"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184268    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6038f7c4-9cd7-445a-bde8-952014f798fb" containerName="istio-proxy"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184274    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184281    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eebe21-8f57-4df3-a107-122eb0d46678" containerName="maas-api"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184286    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184295    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f8622a-90a4-4c6b-898c-dcb95241545e" containerName="payload-processing"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184299    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50937ef-fa83-4104-837a-01a5f003023b" containerName="maas-api"
Oct 31 18:56:02.184589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.184304    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="66279707-0137-4a71-87fb-da0938885850" containerName="payload-pre-processing"
Oct 31 18:56:02.189013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.188993    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.190713 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.190684    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.190850 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.190822    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6vhk\" (UniqueName: \"kubernetes.io/projected/1aa3b956-def9-40c3-895e-56988eb0abfc-kube-api-access-f6vhk\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.190984 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.190962    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3b956-def9-40c3-895e-56988eb0abfc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.191056 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.191039    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-metrics-certs\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.242206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.242171    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-919b6e-serving-cert\""
Oct 31 18:56:02.246173 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.246140    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"]
Oct 31 18:56:02.291735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.291697    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.291939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.291782    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6vhk\" (UniqueName: \"kubernetes.io/projected/1aa3b956-def9-40c3-895e-56988eb0abfc-kube-api-access-f6vhk\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.291939 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:56:02.291810    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-serving-cert: secret "maas-api-e2e-ipp-b-919b6e-serving-cert" not found
Oct 31 18:56:02.291939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.291833    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3b956-def9-40c3-895e-56988eb0abfc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.291939 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:56:02.291889    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls podName:1aa3b956-def9-40c3-895e-56988eb0abfc nodeName:}" failed. No retries permitted until 2026-10-31 18:56:02.791868636 +0000 UTC m=+2066.023106917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls") pod "maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc" (UID: "1aa3b956-def9-40c3-895e-56988eb0abfc") : secret "maas-api-e2e-ipp-b-919b6e-serving-cert" not found
Oct 31 18:56:02.291939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.291921    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-metrics-certs\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.292564 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.292531    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3b956-def9-40c3-895e-56988eb0abfc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.294863 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.294842    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-metrics-certs\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.349010 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.348982    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6vhk\" (UniqueName: \"kubernetes.io/projected/1aa3b956-def9-40c3-895e-56988eb0abfc-kube-api-access-f6vhk\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.796930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.796897    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.799410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.799384    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.805292 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.805269    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:02.909559 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:02.909526    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50937ef-fa83-4104-837a-01a5f003023b" path="/var/lib/kubelet/pods/b50937ef-fa83-4104-837a-01a5f003023b/volumes"
Oct 31 18:56:03.186062 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:03.186037    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"]
Oct 31 18:56:03.187685 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:56:03.187658    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa3b956_def9_40c3_895e_56988eb0abfc.slice/crio-ec7dd9d41510e2f12d669c25d05a6ae78b25c983f3371311b739955cf6e10d78 WatchSource:0}: Error finding container ec7dd9d41510e2f12d669c25d05a6ae78b25c983f3371311b739955cf6e10d78: Status 404 returned error can't find the container with id ec7dd9d41510e2f12d669c25d05a6ae78b25c983f3371311b739955cf6e10d78
Oct 31 18:56:03.271311 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:03.271281    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc" event={"ID":"1aa3b956-def9-40c3-895e-56988eb0abfc","Type":"ContainerStarted","Data":"ec7dd9d41510e2f12d669c25d05a6ae78b25c983f3371311b739955cf6e10d78"}
Oct 31 18:56:04.280506 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:04.280260    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc" event={"ID":"1aa3b956-def9-40c3-895e-56988eb0abfc","Type":"ContainerStarted","Data":"b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5"}
Oct 31 18:56:04.281000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:04.280677    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:04.396198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:04.396139    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc" podStartSLOduration=1.5642449809999999 podStartE2EDuration="2.396124619s" totalImagesPullingTime="831.879638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:04.393590411 +0000 UTC m=+2067.624828703" watchObservedRunningTime="2026-10-31 18:56:04.396124619 +0000 UTC m=+2067.627362909"
Oct 31 18:56:10.294415 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:10.294384    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 18:56:10.312095 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:10.312057    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"]
Oct 31 18:56:10.312264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:10.312244    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy" containerID="cri-o://50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41" gracePeriod=30
Oct 31 18:56:15.615051 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.615023    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:56:15.649216 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649192    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-certs\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.649376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649275    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-socket\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.649376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649303    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-credential-socket\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.649376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649331    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-kube-api-access-5k4dw\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-kube-api-access-5k4dw\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.649579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649518    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-podinfo\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.649579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649524    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-certs" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:15.649579 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649566    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c002e88e-f508-445a-8b51-4436e5b4b93b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002e88e-f508-445a-8b51-4436e5b4b93b-istiod-ca-cert\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.649744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649593    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-data\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.649744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649622    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-envoy\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.649744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.649650    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-token\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-token\") pod \"c002e88e-f508-445a-8b51-4436e5b4b93b\" (UID: \"c002e88e-f508-445a-8b51-4436e5b4b93b\") "
Oct 31 18:56:15.650156 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.650129    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:15.674525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.659531    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-credential-socket" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:15.674525 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.671497    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-kube-api-access-5k4dw" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "kube-api-access-5k4dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:15.674727 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.674584    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-socket" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:15.675075 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.675049    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c002e88e-f508-445a-8b51-4436e5b4b93b-istiod-ca-cert" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:15.675383 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.675346    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-data" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:15.677480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.677426    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-podinfo" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:56:15.677594 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.677573    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-token" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:15.678215 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.678196    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-envoy" pod "c002e88e-f508-445a-8b51-4436e5b4b93b" (UID: "c002e88e-f508-445a-8b51-4436e5b4b93b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:15.774482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.773557    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:15.774482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.773587    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:15.774482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.773604    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:15.774482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.773619    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k4dw\" (UniqueName: \"kubernetes.io/projected/c002e88e-f508-445a-8b51-4436e5b4b93b-kube-api-access-5k4dw\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:15.774482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.773634    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:15.774482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.773648    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002e88e-f508-445a-8b51-4436e5b4b93b-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:15.774482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.773661    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:15.774482 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:15.773675    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002e88e-f508-445a-8b51-4436e5b4b93b-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:16.386196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.386159    5551 generic.go:356] "Generic (PLEG): container finished" podID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerID="50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41" exitCode=0
Oct 31 18:56:16.386382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.386223    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"
Oct 31 18:56:16.386382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.386245    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" event={"ID":"c002e88e-f508-445a-8b51-4436e5b4b93b","Type":"ContainerDied","Data":"50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41"}
Oct 31 18:56:16.386382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.386307    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6" event={"ID":"c002e88e-f508-445a-8b51-4436e5b4b93b","Type":"ContainerDied","Data":"acf77b87dc5818a8e4f6729b03ace1a9d2d093e7f43b81b75c09ec0fa9ebe9a8"}
Oct 31 18:56:16.386382 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.386330    5551 scope.go:122] "RemoveContainer" containerID="50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41"
Oct 31 18:56:16.397342 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.397323    5551 scope.go:122] "RemoveContainer" containerID="50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41"
Oct 31 18:56:16.397752 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:56:16.397728    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41\": container with ID starting with 50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41 not found: ID does not exist" containerID="50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41"
Oct 31 18:56:16.397830 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.397765    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41"} err="failed to get container status \"50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41\": rpc error: code = NotFound desc = could not find container \"50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41\": container with ID starting with 50834b4ba99798a7dbd41ea05cfe39e8f98f8f632eab107002e515a7744dfa41 not found: ID does not exist"
Oct 31 18:56:16.564551 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.564518    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"]
Oct 31 18:56:16.603180 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.603137    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-lmxg6"]
Oct 31 18:56:16.619114 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:56:16.619073    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:56:16.909161 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:16.909127    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" path="/var/lib/kubelet/pods/c002e88e-f508-445a-8b51-4436e5b4b93b/volumes"
Oct 31 18:56:19.932721 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:19.932686    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"]
Oct 31 18:56:19.933125 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:19.932884    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" containerID="cri-o://49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57" gracePeriod=30
Oct 31 18:56:20.162452 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:20.162417    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:56:20.162616 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:20.162509    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:56:24.446532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:24.446503    5551 generic.go:356] "Generic (PLEG): container finished" podID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerID="7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7" exitCode=0
Oct 31 18:56:24.446919 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:24.446545    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" event={"ID":"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d","Type":"ContainerDied","Data":"7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7"}
Oct 31 18:56:24.446985 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:24.446917    5551 scope.go:122] "RemoveContainer" containerID="7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7"
Oct 31 18:56:24.448228 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:24.448199    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:24.448303 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:24.448269    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:24.448340 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:24.448302    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:25.247210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.247187    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:56:25.330412 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330382    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-kube-api-access-gq9bw\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-kube-api-access-gq9bw\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.330623 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330434    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-token\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-token\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.330623 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330525    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355150cb-d1ce-4ff4-8c0c-d490ff467821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355150cb-d1ce-4ff4-8c0c-d490ff467821-istiod-ca-cert\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.330755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330644    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-certs\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.330755 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330725    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-envoy\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.330862 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330783    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-socket\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.330862 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330817    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-data\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.330862 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330851    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-credential-socket\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.331024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330884    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-podinfo\") pod \"355150cb-d1ce-4ff4-8c0c-d490ff467821\" (UID: \"355150cb-d1ce-4ff4-8c0c-d490ff467821\") "
Oct 31 18:56:25.331024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330931    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-certs" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:25.331024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.330955    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355150cb-d1ce-4ff4-8c0c-d490ff467821-istiod-ca-cert" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:25.331172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.331140    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-socket" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:25.331235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.331161    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-data" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:25.331235 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.331176    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-credential-socket" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:25.331689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.331551    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355150cb-d1ce-4ff4-8c0c-d490ff467821-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.331689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.331578    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.331689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.331593    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.331689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.331607    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.331689 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.331623    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.333258 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.333234    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-envoy" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:25.333258 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.333247    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-podinfo" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:56:25.333546 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.333526    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-token" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:25.333617 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.333545    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-kube-api-access-gq9bw" pod "355150cb-d1ce-4ff4-8c0c-d490ff467821" (UID: "355150cb-d1ce-4ff4-8c0c-d490ff467821"). InnerVolumeSpecName "kube-api-access-gq9bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:25.432701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.432668    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.432701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.432698    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.432701 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.432708    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9bw\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-kube-api-access-gq9bw\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.432928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.432717    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355150cb-d1ce-4ff4-8c0c-d490ff467821-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:25.485992 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.485890    5551 generic.go:356] "Generic (PLEG): container finished" podID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerID="49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57" exitCode=0
Oct 31 18:56:25.485992 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.485940    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"
Oct 31 18:56:25.486585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.485986    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" event={"ID":"355150cb-d1ce-4ff4-8c0c-d490ff467821","Type":"ContainerDied","Data":"49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57"}
Oct 31 18:56:25.486585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.486081    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9" event={"ID":"355150cb-d1ce-4ff4-8c0c-d490ff467821","Type":"ContainerDied","Data":"b67014a37704b0c8f09ea791e4546bbf94ca45e33c322d137f14448c37a2e3d5"}
Oct 31 18:56:25.486585 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.486112    5551 scope.go:122] "RemoveContainer" containerID="49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57"
Oct 31 18:56:25.488719 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.488549    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" event={"ID":"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d","Type":"ContainerStarted","Data":"148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67"}
Oct 31 18:56:25.497752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.497735    5551 scope.go:122] "RemoveContainer" containerID="49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57"
Oct 31 18:56:25.498076 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:56:25.498026    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57\": container with ID starting with 49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57 not found: ID does not exist" containerID="49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57"
Oct 31 18:56:25.498076 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.498061    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57"} err="failed to get container status \"49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57\": rpc error: code = NotFound desc = could not find container \"49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57\": container with ID starting with 49c9e0475fda81d514882df5d527fb5b0ef6150bb82d0b8587149f27aaf11a57 not found: ID does not exist"
Oct 31 18:56:25.796542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.787958    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"]
Oct 31 18:56:25.802873 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.802830    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-8rnf9"]
Oct 31 18:56:25.845096 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.845061    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:56:25.846172 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.846144    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 31 18:56:25.846249 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:25.846199    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 31 18:56:26.845445 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:26.845413    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 31 18:56:26.845839 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:26.845483    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 31 18:56:26.910589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:26.910557    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" path="/var/lib/kubelet/pods/355150cb-d1ce-4ff4-8c0c-d490ff467821/volumes"
Oct 31 18:56:27.285639 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.285566    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"]
Oct 31 18:56:27.287088 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.286650    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy"
Oct 31 18:56:27.287300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287249    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy"
Oct 31 18:56:27.287300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287295    5551 cpu_manager.go:414] "containerMap: removing container" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy"
Oct 31 18:56:27.287300 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287305    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040453dc-9c8e-4741-a823-398aa8e3bc61" containerName="istio-proxy"
Oct 31 18:56:27.287660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287320    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy"
Oct 31 18:56:27.287660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287327    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e02db3a-b339-4d5c-b8f0-b1ea23f6b705" containerName="istio-proxy"
Oct 31 18:56:27.287660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287336    5551 cpu_manager.go:414] "containerMap: removing container" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy"
Oct 31 18:56:27.287660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287343    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy"
Oct 31 18:56:27.287660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287565    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c002e88e-f508-445a-8b51-4436e5b4b93b" containerName="istio-proxy"
Oct 31 18:56:27.287660 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.287582    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="355150cb-d1ce-4ff4-8c0c-d490ff467821" containerName="istio-proxy"
Oct 31 18:56:27.293790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.293768    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.368879 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.368826    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-35566a-openshift-default-dockercfg-mlv78\""
Oct 31 18:56:27.387433 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.387318    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4jhp\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-kube-api-access-v4jhp\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.387650 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.387501    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-socket\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.391696 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.391659    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-credential-socket\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.391849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.391836    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-token\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.392007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.391982    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-data\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.392112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.392074    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-envoy\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.392224 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.392186    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istiod-ca-cert\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.392292 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.392258    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-certs\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.392369 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.392290    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-podinfo\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.426193 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.426158    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"]
Oct 31 18:56:27.498560 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.498489    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4jhp\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-kube-api-access-v4jhp\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.498820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.498572    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-socket\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.498820 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.498745    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-credential-socket\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.498987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.498838    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-token\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.498987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.498907    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-data\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.498987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.498941    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-envoy\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.498987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.498981    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istiod-ca-cert\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.499285 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.499028    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-certs\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.499285 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.499056    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-podinfo\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.499285 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.499270    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-socket\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.499556 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.499537    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-data\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.500182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.499712    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-credential-socket\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.500182 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.500144    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-certs\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.504480 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.502078    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istiod-ca-cert\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.508523 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.508488    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-podinfo\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.509673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.509623    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-envoy\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.570528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.566920    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-token\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.570693 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.570522    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4jhp\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-kube-api-access-v4jhp\") pod \"e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") " pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.618951 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.618900    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:27.846134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.845648    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 31 18:56:27.846134 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:27.845712    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 31 18:56:28.093473 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.093427    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"]
Oct 31 18:56:28.095668 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:56:28.095636    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed65c6e_fc1f_4857_9ad0_2ebfd1d56e35.slice/crio-629df6261f0c05dbea9d4c428ccb215123b13e367e0566936e98399bb324c082 WatchSource:0}: Error finding container 629df6261f0c05dbea9d4c428ccb215123b13e367e0566936e98399bb324c082: Status 404 returned error can't find the container with id 629df6261f0c05dbea9d4c428ccb215123b13e367e0566936e98399bb324c082
Oct 31 18:56:28.098902 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.098869    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:28.098993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.098936    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:28.098993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.098965    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:28.518410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.518364    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" event={"ID":"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35","Type":"ContainerStarted","Data":"a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5"}
Oct 31 18:56:28.518602 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.518437    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" event={"ID":"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35","Type":"ContainerStarted","Data":"629df6261f0c05dbea9d4c428ccb215123b13e367e0566936e98399bb324c082"}
Oct 31 18:56:28.620001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.619963    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:28.638029 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.637969    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" podStartSLOduration=1.6379489820000002 podStartE2EDuration="1.637948982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:28.635678636 +0000 UTC m=+2091.866916926" watchObservedRunningTime="2026-10-31 18:56:28.637948982 +0000 UTC m=+2091.869187275"
Oct 31 18:56:28.845476 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.845380    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 31 18:56:28.845618 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:28.845484    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 31 18:56:29.372484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.369071    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"]
Oct 31 18:56:29.389152 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.389122    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.428176 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.428141    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"]
Oct 31 18:56:29.443222 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.443189    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4157da-openshift-default-dockercfg-82z8j\""
Oct 31 18:56:29.523511 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523445    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-podinfo\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.523674 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523596    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-token\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.523674 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523655    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvjbc\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-kube-api-access-xvjbc\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.523764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523678    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-data\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.523807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523768    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-socket\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.523843 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523809    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-envoy\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.523885 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523841    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-certs\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.523930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523888    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-credential-socket\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.524013 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.523994    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a940ce0-f138-492c-84b6-aaf10d45a459-istiod-ca-cert\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.624748 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.624722    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-credential-socket\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.624928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.624822    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a940ce0-f138-492c-84b6-aaf10d45a459-istiod-ca-cert\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.624928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.624861    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-podinfo\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.624928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.624902    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-token\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625123 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.624959    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvjbc\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-kube-api-access-xvjbc\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625123 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.624989    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-data\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625123 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625034    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-socket\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625123 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625062    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-envoy\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625123 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625109    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-certs\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625410 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625135    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-credential-socket\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625568    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-socket\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625711 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625612    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a940ce0-f138-492c-84b6-aaf10d45a459-istiod-ca-cert\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.625777 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625724    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:29.625904 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625887    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-data\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.626014 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.625925    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-certs\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.627781 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.627750    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-podinfo\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.627868 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.627854    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-envoy\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.685256 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.685217    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvjbc\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-kube-api-access-xvjbc\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.686137 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.686087    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-token\") pod \"e2e-praxis-a-4157da-openshift-default-679dccb777-pztct\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") " pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.723001 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.722901    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:56:29.723401 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.723381    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:29.845002 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.844970    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 31 18:56:29.845167 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:29.845034    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 31 18:56:30.203236 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.203150    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"]
Oct 31 18:56:30.205925 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:56:30.205897    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a940ce0_f138_492c_84b6_aaf10d45a459.slice/crio-bea2321e8cfe762ec6ae6c5014ed4bdf0249b2888de42a728e34ee4524cab456 WatchSource:0}: Error finding container bea2321e8cfe762ec6ae6c5014ed4bdf0249b2888de42a728e34ee4524cab456: Status 404 returned error can't find the container with id bea2321e8cfe762ec6ae6c5014ed4bdf0249b2888de42a728e34ee4524cab456
Oct 31 18:56:30.208493 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.208440    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:30.208610 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.208527    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:30.208610 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.208553    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:30.537524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.537288    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" event={"ID":"0a940ce0-f138-492c-84b6-aaf10d45a459","Type":"ContainerStarted","Data":"aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f"}
Oct 31 18:56:30.537524 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.537359    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" event={"ID":"0a940ce0-f138-492c-84b6-aaf10d45a459","Type":"ContainerStarted","Data":"bea2321e8cfe762ec6ae6c5014ed4bdf0249b2888de42a728e34ee4524cab456"}
Oct 31 18:56:30.537912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.537656    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:30.538595 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.538556    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:56:30.693495 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.691655    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" podStartSLOduration=1.6916331740000001 podStartE2EDuration="1.691633174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:30.65778211 +0000 UTC m=+2093.889020414" watchObservedRunningTime="2026-10-31 18:56:30.691633174 +0000 UTC m=+2093.922871465"
Oct 31 18:56:30.723728 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.723695    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:30.845437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.845358    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 31 18:56:30.845437 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:30.845421    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 31 18:56:31.546244 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.546216    5551 generic.go:356] "Generic (PLEG): container finished" podID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerID="484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06" exitCode=0
Oct 31 18:56:31.546653 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.546290    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" event={"ID":"7999f6f9-a7e8-4ff7-a9de-57aefb668280","Type":"ContainerDied","Data":"484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06"}
Oct 31 18:56:31.546922 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.546903    5551 scope.go:122] "RemoveContainer" containerID="484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06"
Oct 31 18:56:31.548122 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.548097    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:31.548177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.548167    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:31.548213 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.548199    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:31.731024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.730957    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:31.851139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.851059    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:56:31.852264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:31.852237    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 18:56:32.615485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:32.610330    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" event={"ID":"7999f6f9-a7e8-4ff7-a9de-57aefb668280","Type":"ContainerStarted","Data":"835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841"}
Oct 31 18:56:32.616104 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:32.616072    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:32.616847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:32.616822    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:56:33.260484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:33.256831    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:56:33.260484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:33.256954    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:56:33.260484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:33.257006    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 31 18:56:34.268485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:34.262875    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:56:34.268485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:34.262946    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 31 18:56:35.275491 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:35.274873    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:56:35.275491 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:35.274936    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 31 18:56:36.262898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:36.262861    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:56:36.263177 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:36.263146    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 31 18:56:37.261484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:37.259856    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:56:37.261484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:37.259920    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 31 18:56:38.260734 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:38.260702    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:56:38.261087 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:38.261065    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:56:38.261772 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:38.261756    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:56:39.744202 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.744162    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"]
Oct 31 18:56:39.750108 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.750078    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823172    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-credential-socket\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823207 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823210    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc4hn\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-kube-api-access-qc4hn\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823418 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823232    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-socket\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823418 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823386    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba30e08-8922-4c58-bf01-812abde01f55-istio-podinfo\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823532 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823447    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-istio-token\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823592 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823574    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-certs\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823699 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823683    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-envoy\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823787 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823775    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-data\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.823827 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.823808    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba30e08-8922-4c58-bf01-812abde01f55-istiod-ca-cert\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.841729 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.841693    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3932ef-openshift-default-dockercfg-gmhq9\""
Oct 31 18:56:39.855947 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.855915    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"]
Oct 31 18:56:39.924756 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.924723    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-data\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.924918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.924765    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba30e08-8922-4c58-bf01-812abde01f55-istiod-ca-cert\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.924918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.924799    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-credential-socket\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.924918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.924816    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4hn\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-kube-api-access-qc4hn\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.924918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.924835    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-socket\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.924918 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.924857    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba30e08-8922-4c58-bf01-812abde01f55-istio-podinfo\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.925150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.924969    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-istio-token\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.925150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.925053    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-certs\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.925150 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.925139    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-envoy\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.925314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.925160    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-data\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.925314 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.925184    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-credential-socket\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.925520 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.925499    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-certs\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.925596 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.925569    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-socket\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.925642 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.925627    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba30e08-8922-4c58-bf01-812abde01f55-istiod-ca-cert\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.927472 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.927432    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba30e08-8922-4c58-bf01-812abde01f55-istio-podinfo\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.927571 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.927537    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-envoy\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.986183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.986123    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-istio-token\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:39.986431 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:39.986405    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4hn\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-kube-api-access-qc4hn\") pod \"e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") " pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:40.068301 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.066370    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:40.104987 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.104954    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:56:40.115522 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.115494    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 18:56:40.515124 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.515101    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"]
Oct 31 18:56:40.517108 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:56:40.517080    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba30e08_8922_4c58_bf01_812abde01f55.slice/crio-cabef3107507446af82f3b6e8c3c68f410e72366d8d62c4ff010af9e8978ab8d WatchSource:0}: Error finding container cabef3107507446af82f3b6e8c3c68f410e72366d8d62c4ff010af9e8978ab8d: Status 404 returned error can't find the container with id cabef3107507446af82f3b6e8c3c68f410e72366d8d62c4ff010af9e8978ab8d
Oct 31 18:56:40.519848 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.519814    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:40.519935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.519881    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:40.519935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.519907    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:40.664048 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.664012    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" event={"ID":"5ba30e08-8922-4c58-bf01-812abde01f55","Type":"ContainerStarted","Data":"aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e"}
Oct 31 18:56:40.664262 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.664074    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" event={"ID":"5ba30e08-8922-4c58-bf01-812abde01f55","Type":"ContainerStarted","Data":"cabef3107507446af82f3b6e8c3c68f410e72366d8d62c4ff010af9e8978ab8d"}
Oct 31 18:56:40.784601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:40.784487    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podStartSLOduration=1.7844473220000001 podStartE2EDuration="1.784447322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:40.779560476 +0000 UTC m=+2104.010798767" watchObservedRunningTime="2026-10-31 18:56:40.784447322 +0000 UTC m=+2104.015685612"
Oct 31 18:56:41.068043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:41.067950    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:41.075043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:41.075010    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:41.672175 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:41.672145    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:41.673210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:41.673180    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:56:46.605164 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:56:46.605122    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:56:58.420145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.420104    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"]
Oct 31 18:56:58.424090 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.424071    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.490547 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.490502    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-713bf0-openshift-default-dockercfg-vtdcw\""
Oct 31 18:56:58.536877 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.536845    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"]
Oct 31 18:56:58.564183 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564155    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-envoy\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.564345 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564189    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-podinfo\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.564345 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564213    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-data\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.564345 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564230    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-credential-socket\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.564511 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564370    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320c4562-4bfc-4db1-8839-068ae02a7dc9-istiod-ca-cert\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.564556 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564524    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-token\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.564601 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564578    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-socket\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.564657 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564603    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7jk\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-kube-api-access-cx7jk\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.564731 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.564711    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-certs\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.665567 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.665537    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-data\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.665567 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.665573    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-credential-socket\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.665831 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.665761    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320c4562-4bfc-4db1-8839-068ae02a7dc9-istiod-ca-cert\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.665898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.665880    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-token\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.665947    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-socket\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666024 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.665987    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7jk\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-kube-api-access-cx7jk\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.666015    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-data\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.666026    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-certs\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.666048    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-credential-socket\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.666070    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-envoy\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666139 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.666114    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-podinfo\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666388 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.666377    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320c4562-4bfc-4db1-8839-068ae02a7dc9-istiod-ca-cert\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666445 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.666424    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-socket\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.666549 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.666529    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-certs\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.668553 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.668528    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-envoy\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.671210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.671162    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-podinfo\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.726272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.725115    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-token\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.726997 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.726973    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7jk\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-kube-api-access-cx7jk\") pod \"e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") " pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.738829 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.738806    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:56:58.788815 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.788785    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"]
Oct 31 18:56:58.897952 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:58.897918    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"]
Oct 31 18:56:59.170264 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:59.170238    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"]
Oct 31 18:56:59.171520 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:56:59.171492    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320c4562_4bfc_4db1_8839_068ae02a7dc9.slice/crio-36e8857f843fa47db6bacffc4d144db79dbd5e3e4b116c7c42ca93396beb747a WatchSource:0}: Error finding container 36e8857f843fa47db6bacffc4d144db79dbd5e3e4b116c7c42ca93396beb747a: Status 404 returned error can't find the container with id 36e8857f843fa47db6bacffc4d144db79dbd5e3e4b116c7c42ca93396beb747a
Oct 31 18:56:59.173930 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:59.173898    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:59.174015 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:59.173967    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:59.174015 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:59.173993    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:56:59.792248 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:59.792200    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" event={"ID":"320c4562-4bfc-4db1-8839-068ae02a7dc9","Type":"ContainerStarted","Data":"8de1d6332ee4dd0b1b1e2ff8b0cbcbcb04ea109681639b861e0f928c8218217e"}
Oct 31 18:56:59.792804 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:59.792274    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" event={"ID":"320c4562-4bfc-4db1-8839-068ae02a7dc9","Type":"ContainerStarted","Data":"36e8857f843fa47db6bacffc4d144db79dbd5e3e4b116c7c42ca93396beb747a"}
Oct 31 18:56:59.912271 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:56:59.912201    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" podStartSLOduration=1.9121812280000001 podStartE2EDuration="1.912181228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:59.908813415 +0000 UTC m=+2123.140051706" watchObservedRunningTime="2026-10-31 18:56:59.912181228 +0000 UTC m=+2123.143419527"
Oct 31 18:57:00.739414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:00.739369    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:57:00.745066 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:00.745038    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:57:00.800450 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:00.800412    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:57:00.801539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:00.801518    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:57:03.789988 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:03.789938    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw" podUID="b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5" containerName="payload-processing" containerID="cri-o://d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e" gracePeriod=25
Oct 31 18:57:03.899010 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:03.898962    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc" podUID="97a91f0c-6869-4205-ae00-8cfca222d370" containerName="payload-pre-processing" containerID="cri-o://84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef" gracePeriod=25
Oct 31 18:57:04.072747 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.072721    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:57:04.132639 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.132605    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-kube-api-access-85xl7\" (UniqueName: \"kubernetes.io/projected/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-kube-api-access-85xl7\") pod \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\" (UID: \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\") "
Oct 31 18:57:04.132802 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.132760    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-plugins-config-volume\") pod \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\" (UID: \"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5\") "
Oct 31 18:57:04.133667 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.133605    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-plugins-config-volume" pod "b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5" (UID: "b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:57:04.140607 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.140538    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-kube-api-access-85xl7" pod "b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5" (UID: "b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5"). InnerVolumeSpecName "kube-api-access-85xl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:57:04.166415 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.166390    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:57:04.234307 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.234268    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a91f0c-6869-4205-ae00-8cfca222d370-kube-api-access-76mbm\" (UniqueName: \"kubernetes.io/projected/97a91f0c-6869-4205-ae00-8cfca222d370-kube-api-access-76mbm\") pod \"97a91f0c-6869-4205-ae00-8cfca222d370\" (UID: \"97a91f0c-6869-4205-ae00-8cfca222d370\") "
Oct 31 18:57:04.234528 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.234389    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a91f0c-6869-4205-ae00-8cfca222d370-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a91f0c-6869-4205-ae00-8cfca222d370-plugins-config-volume\") pod \"97a91f0c-6869-4205-ae00-8cfca222d370\" (UID: \"97a91f0c-6869-4205-ae00-8cfca222d370\") "
Oct 31 18:57:04.234790 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.234766    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a91f0c-6869-4205-ae00-8cfca222d370-plugins-config-volume" pod "97a91f0c-6869-4205-ae00-8cfca222d370" (UID: "97a91f0c-6869-4205-ae00-8cfca222d370"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:57:04.234938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.234920    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85xl7\" (UniqueName: \"kubernetes.io/projected/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-kube-api-access-85xl7\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:57:04.234990 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.234942    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a91f0c-6869-4205-ae00-8cfca222d370-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:57:04.234990 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.234953    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:57:04.236837 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.236818    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a91f0c-6869-4205-ae00-8cfca222d370-kube-api-access-76mbm" pod "97a91f0c-6869-4205-ae00-8cfca222d370" (UID: "97a91f0c-6869-4205-ae00-8cfca222d370"). InnerVolumeSpecName "kube-api-access-76mbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:57:04.336759 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.336678    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76mbm\" (UniqueName: \"kubernetes.io/projected/97a91f0c-6869-4205-ae00-8cfca222d370-kube-api-access-76mbm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:57:04.827271 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.827237    5551 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5" containerID="d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e" exitCode=0
Oct 31 18:57:04.827702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.827311    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"
Oct 31 18:57:04.827702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.827317    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw" event={"ID":"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5","Type":"ContainerDied","Data":"d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e"}
Oct 31 18:57:04.827702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.827391    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw" event={"ID":"b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5","Type":"ContainerDied","Data":"d58b297637d7bed77d02f7de6bb5b1c8c81b8af84685b228979f4b12ea2fe7ea"}
Oct 31 18:57:04.827702 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.827414    5551 scope.go:122] "RemoveContainer" containerID="d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e"
Oct 31 18:57:04.828819 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.828799    5551 generic.go:356] "Generic (PLEG): container finished" podID="97a91f0c-6869-4205-ae00-8cfca222d370" containerID="84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef" exitCode=0
Oct 31 18:57:04.828939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.828840    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc" event={"ID":"97a91f0c-6869-4205-ae00-8cfca222d370","Type":"ContainerDied","Data":"84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef"}
Oct 31 18:57:04.828939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.828863    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"
Oct 31 18:57:04.828939 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.828873    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc" event={"ID":"97a91f0c-6869-4205-ae00-8cfca222d370","Type":"ContainerDied","Data":"6f0c92bd9c3e0a25a1f0021ce64f2a748e2ae33a8b97bce98524fe0824346258"}
Oct 31 18:57:04.838335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.838233    5551 scope.go:122] "RemoveContainer" containerID="d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e"
Oct 31 18:57:04.838697 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:57:04.838668    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e\": container with ID starting with d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e not found: ID does not exist" containerID="d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e"
Oct 31 18:57:04.838807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.838709    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e"} err="failed to get container status \"d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e\": rpc error: code = NotFound desc = could not find container \"d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e\": container with ID starting with d7b1967410a9cdff475fa647070d131159d088efff28af94a7e344a15fa6cc0e not found: ID does not exist"
Oct 31 18:57:04.838807 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.838734    5551 scope.go:122] "RemoveContainer" containerID="84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef"
Oct 31 18:57:04.850349 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.850328    5551 scope.go:122] "RemoveContainer" containerID="84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef"
Oct 31 18:57:04.850801 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:57:04.850774    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef\": container with ID starting with 84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef not found: ID does not exist" containerID="84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef"
Oct 31 18:57:04.850922 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.850811    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef"} err="failed to get container status \"84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef\": rpc error: code = NotFound desc = could not find container \"84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef\": container with ID starting with 84ae0d87df81a20089c4e21c666bb81ff8d60396d17c01a2eb9af9ad6a6f5eef not found: ID does not exist"
Oct 31 18:57:04.995818 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:04.995759    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"]
Oct 31 18:57:05.045421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:05.045385    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6d84d8f7f-c7mbc"]
Oct 31 18:57:05.112143 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:05.112057    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"]
Oct 31 18:57:05.114651 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:05.114632    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-919b6e-58c49d7c55-2w8qw"]
Oct 31 18:57:06.908141 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:06.908110    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a91f0c-6869-4205-ae00-8cfca222d370" path="/var/lib/kubelet/pods/97a91f0c-6869-4205-ae00-8cfca222d370/volumes"
Oct 31 18:57:06.908544 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:06.908530    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5" path="/var/lib/kubelet/pods/b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5/volumes"
Oct 31 18:57:07.800526 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.800485    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"]
Oct 31 18:57:07.801043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.801028    5551 cpu_manager.go:414] "containerMap: removing container" podUID="97a91f0c-6869-4205-ae00-8cfca222d370" containerName="payload-pre-processing"
Oct 31 18:57:07.801120 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.801043    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a91f0c-6869-4205-ae00-8cfca222d370" containerName="payload-pre-processing"
Oct 31 18:57:07.801120 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.801064    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5" containerName="payload-processing"
Oct 31 18:57:07.801120 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.801070    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5" containerName="payload-processing"
Oct 31 18:57:07.801277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.801148    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a91f0c-6869-4205-ae00-8cfca222d370" containerName="payload-pre-processing"
Oct 31 18:57:07.801277 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.801159    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e1ae25-63b2-4ac0-95a1-9b7fcbbfc2e5" containerName="payload-processing"
Oct 31 18:57:07.806043 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.806017    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:07.860993 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.860935    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"]
Oct 31 18:57:07.921484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.900734    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk\" (UID: \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:07.921484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.900934    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl9db\" (UniqueName: \"kubernetes.io/projected/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-kube-api-access-gl9db\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk\" (UID: \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:07.939914 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.927806    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-713bf0\""
Oct 31 18:57:07.939914 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:07.927989    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-713bf0-dockercfg-nj2k5\""
Oct 31 18:57:08.001744 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.001709    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl9db\" (UniqueName: \"kubernetes.io/projected/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-kube-api-access-gl9db\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk\" (UID: \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:08.001943 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.001864    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk\" (UID: \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:08.002804 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.002780    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk\" (UID: \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:08.058346 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.058273    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl9db\" (UniqueName: \"kubernetes.io/projected/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-kube-api-access-gl9db\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk\" (UID: \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:08.120076 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.120048    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:08.499112 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.499080    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"]
Oct 31 18:57:08.501265 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:57:08.501233    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef3112e_e1cd_4899_8af7_d2188fcb2e40.slice/crio-44849b5cac60c0db1389cfd87e987c2b3d4fa4c046c8cb99a90f3c1c99ab2f06 WatchSource:0}: Error finding container 44849b5cac60c0db1389cfd87e987c2b3d4fa4c046c8cb99a90f3c1c99ab2f06: Status 404 returned error can't find the container with id 44849b5cac60c0db1389cfd87e987c2b3d4fa4c046c8cb99a90f3c1c99ab2f06
Oct 31 18:57:08.861374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.861276    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk" event={"ID":"4ef3112e-e1cd-4899-8af7-d2188fcb2e40","Type":"ContainerStarted","Data":"3d429f0cfbfbcba82f5b0088b89773d1bf72a73e697276d611a46e73e63a79f8"}
Oct 31 18:57:08.861374 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.861339    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk" event={"ID":"4ef3112e-e1cd-4899-8af7-d2188fcb2e40","Type":"ContainerStarted","Data":"44849b5cac60c0db1389cfd87e987c2b3d4fa4c046c8cb99a90f3c1c99ab2f06"}
Oct 31 18:57:08.861598 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.861489    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:08.908326 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.908298    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"]
Oct 31 18:57:08.912234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.912215    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:08.964506 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.964474    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-676224\"/\"kube-root-ca.crt\""
Oct 31 18:57:08.964506 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.964506    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-676224\"/\"ipp-route-676224-kserve-self-signed-certs\""
Oct 31 18:57:08.964961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.964561    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-676224\"/\"default-dockercfg-k9kvc\""
Oct 31 18:57:08.964961 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.964595    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-676224\"/\"openshift-service-ca.crt\""
Oct 31 18:57:08.965849 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:08.965825    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"]
Oct 31 18:57:09.012376 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.012346    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-home\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.012541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.012407    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tmp-dir\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.012541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.012535    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-dshm\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.012630 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.012580    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tls-certs\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.012705 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.012690    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrx7\" (UniqueName: \"kubernetes.io/projected/0eb97b27-3e63-4ef2-bad2-faebae4ac636-kube-api-access-gkrx7\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.012752 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.012717    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-model-cache\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.106483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.106043    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk" podStartSLOduration=2.106021822 podStartE2EDuration="2.106021822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:57:08.977650907 +0000 UTC m=+2132.208889199" watchObservedRunningTime="2026-10-31 18:57:09.106021822 +0000 UTC m=+2132.337260113"
Oct 31 18:57:09.114536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.114355    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-home\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.114536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.114451    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tmp-dir\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.114536 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.114505    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-dshm\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.114822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.114555    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tls-certs\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.114822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.114672    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrx7\" (UniqueName: \"kubernetes.io/projected/0eb97b27-3e63-4ef2-bad2-faebae4ac636-kube-api-access-gkrx7\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.114822 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.114703    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-model-cache\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.115191 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.115156    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-model-cache\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.115428 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.115405    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-home\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.115677 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.115653    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tmp-dir\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.118557 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.118529    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-dshm\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.119010 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.118980    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tls-certs\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.177957 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.177925    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrx7\" (UniqueName: \"kubernetes.io/projected/0eb97b27-3e63-4ef2-bad2-faebae4ac636-kube-api-access-gkrx7\") pod \"ipp-route-676224-kserve-699b8c644f-dvrv7\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") " pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.224126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.224091    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:09.601898 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.601857    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"]
Oct 31 18:57:09.602947 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:57:09.602917    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eb97b27_3e63_4ef2_bad2_faebae4ac636.slice/crio-48aeb0149b4668ed78f6e2bf6537fbb4b49f40700e5d19dff5cd348c56677f93 WatchSource:0}: Error finding container 48aeb0149b4668ed78f6e2bf6537fbb4b49f40700e5d19dff5cd348c56677f93: Status 404 returned error can't find the container with id 48aeb0149b4668ed78f6e2bf6537fbb4b49f40700e5d19dff5cd348c56677f93
Oct 31 18:57:09.870174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.870072    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7" event={"ID":"0eb97b27-3e63-4ef2-bad2-faebae4ac636","Type":"ContainerStarted","Data":"eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813"}
Oct 31 18:57:09.870174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.870115    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7" event={"ID":"0eb97b27-3e63-4ef2-bad2-faebae4ac636","Type":"ContainerStarted","Data":"48aeb0149b4668ed78f6e2bf6537fbb4b49f40700e5d19dff5cd348c56677f93"}
Oct 31 18:57:09.984366 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:09.984304    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7" podStartSLOduration=1.984285834 podStartE2EDuration="1.984285834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:57:09.982576885 +0000 UTC m=+2133.213815176" watchObservedRunningTime="2026-10-31 18:57:09.984285834 +0000 UTC m=+2133.215524125"
Oct 31 18:57:10.224597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:10.224551    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:16.604906 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:57:16.604863    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:57:19.224915 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:19.224869    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:19.237726 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:19.237701    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:19.873954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:19.873924    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:57:19.950740 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:19.950713    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 18:57:22.537726 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.537691    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"]
Oct 31 18:57:22.541907 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.541886    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.644639 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.644597    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-35566a-serving-cert\""
Oct 31 18:57:22.647753 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.647727    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"]
Oct 31 18:57:22.683005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.682949    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-kube-api-access-mwz7v\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.683209 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.683020    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-metrics-certs\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.683209 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.683184    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.683335 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.683294    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-maas-api-tls\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.785093 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.785054    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-maas-api-tls\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.785310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.785203    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-kube-api-access-mwz7v\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.785310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.785239    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-metrics-certs\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.785310 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.785292    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.786333 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.786300    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.787928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.787855    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-maas-api-tls\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.788181 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.788159    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-metrics-certs\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.842666 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.842626    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-kube-api-access-mwz7v\") pod \"maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:22.857198 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:22.857172    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:23.241788 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:23.241763    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"]
Oct 31 18:57:23.244072 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:57:23.244036    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2fc59f_cd6e_4d4c_a0a4_901e79375b37.slice/crio-f567a85f93bc5673d3e92493e9376f17bd6007c86849177b7b4d40d2efd40d65 WatchSource:0}: Error finding container f567a85f93bc5673d3e92493e9376f17bd6007c86849177b7b4d40d2efd40d65: Status 404 returned error can't find the container with id f567a85f93bc5673d3e92493e9376f17bd6007c86849177b7b4d40d2efd40d65
Oct 31 18:57:24.002870 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:24.002836    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs" event={"ID":"df2fc59f-cd6e-4d4c-a0a4-901e79375b37","Type":"ContainerStarted","Data":"f567a85f93bc5673d3e92493e9376f17bd6007c86849177b7b4d40d2efd40d65"}
Oct 31 18:57:25.012305 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:25.011290    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs" event={"ID":"df2fc59f-cd6e-4d4c-a0a4-901e79375b37","Type":"ContainerStarted","Data":"6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885"}
Oct 31 18:57:25.012305 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:25.011502    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:25.128573 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:25.128520    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs" podStartSLOduration=2.2075855779999998 podStartE2EDuration="3.128505351s" totalImagesPullingTime="920.919773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:57:25.125312693 +0000 UTC m=+2148.356551011" watchObservedRunningTime="2026-10-31 18:57:25.128505351 +0000 UTC m=+2148.359743641"
Oct 31 18:57:27.589531 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.589491    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"]
Oct 31 18:57:27.593435 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.593418    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.648818 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.648795    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4157da-serving-cert\""
Oct 31 18:57:27.652534 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.652507    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"]
Oct 31 18:57:27.661492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.661450    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58zq\" (UniqueName: \"kubernetes.io/projected/6e49e7ca-b485-4395-865d-2f8886165a7e-kube-api-access-t58zq\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.661611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.661563    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e49e7ca-b485-4395-865d-2f8886165a7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.661611 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.661593    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-metrics-certs\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.661767 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.661746    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.762758 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.762722    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58zq\" (UniqueName: \"kubernetes.io/projected/6e49e7ca-b485-4395-865d-2f8886165a7e-kube-api-access-t58zq\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.762920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.762790    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e49e7ca-b485-4395-865d-2f8886165a7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.762920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.762825    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-metrics-certs\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.762920 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.762875    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.763115 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:57:27.763091    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-serving-cert: secret "maas-api-e2e-praxis-a-4157da-serving-cert" not found
Oct 31 18:57:27.763231 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:57:27.763217    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls podName:6e49e7ca-b485-4395-865d-2f8886165a7e nodeName:}" failed. No retries permitted until 2026-10-31 18:57:28.263191626 +0000 UTC m=+2151.494429903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls") pod "maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c" (UID: "6e49e7ca-b485-4395-865d-2f8886165a7e") : secret "maas-api-e2e-praxis-a-4157da-serving-cert" not found
Oct 31 18:57:27.763569 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.763552    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e49e7ca-b485-4395-865d-2f8886165a7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.765331 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.765308    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-metrics-certs\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:27.819815 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:27.819779    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58zq\" (UniqueName: \"kubernetes.io/projected/6e49e7ca-b485-4395-865d-2f8886165a7e-kube-api-access-t58zq\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:28.271839 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:28.271800    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:28.274484 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:28.274434    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:28.507184 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:28.507138    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:28.888169 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:28.888143    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"]
Oct 31 18:57:28.889796 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:57:28.889765    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e49e7ca_b485_4395_865d_2f8886165a7e.slice/crio-800586553f873fe5f562ebf04bda98ba1c02489824c24f1a809f4ff52d14c0f4 WatchSource:0}: Error finding container 800586553f873fe5f562ebf04bda98ba1c02489824c24f1a809f4ff52d14c0f4: Status 404 returned error can't find the container with id 800586553f873fe5f562ebf04bda98ba1c02489824c24f1a809f4ff52d14c0f4
Oct 31 18:57:29.056882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:29.056846    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c" event={"ID":"6e49e7ca-b485-4395-865d-2f8886165a7e","Type":"ContainerStarted","Data":"800586553f873fe5f562ebf04bda98ba1c02489824c24f1a809f4ff52d14c0f4"}
Oct 31 18:57:30.067764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:30.067666    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c" event={"ID":"6e49e7ca-b485-4395-865d-2f8886165a7e","Type":"ContainerStarted","Data":"1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59"}
Oct 31 18:57:30.067764 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:30.067745    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:30.183589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:30.183533    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c" podStartSLOduration=2.358663125 podStartE2EDuration="3.183517982s" totalImagesPullingTime="824.854857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:57:30.181761824 +0000 UTC m=+2153.413000131" watchObservedRunningTime="2026-10-31 18:57:30.183517982 +0000 UTC m=+2153.414756271"
Oct 31 18:57:31.024000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:31.023966    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:57:36.081637 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:36.081605    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 18:57:43.536773 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.536694    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"]
Oct 31 18:57:43.541846 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.541829    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.594269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.594227    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3932ef-serving-cert\""
Oct 31 18:57:43.646797 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.646766    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"]
Oct 31 18:57:43.664048 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.664018    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/b7dc1489-f35d-4533-a74a-796288531f74-kube-api-access-pbm6m\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.664186 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.664159    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.664257 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.664221    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-metrics-certs\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.664313 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.664273    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc1489-f35d-4533-a74a-796288531f74-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.765673 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.765638    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc1489-f35d-4533-a74a-796288531f74-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.765874 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.765773    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/b7dc1489-f35d-4533-a74a-796288531f74-kube-api-access-pbm6m\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.765944 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.765880    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.766000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.765944    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-metrics-certs\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.766446 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.766419    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc1489-f35d-4533-a74a-796288531f74-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.766722 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:57:43.766702    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-serving-cert: secret "maas-api-e2e-praxis-a-3932ef-serving-cert" not found
Oct 31 18:57:43.766895 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:57:43.766882    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls podName:b7dc1489-f35d-4533-a74a-796288531f74 nodeName:}" failed. No retries permitted until 2026-10-31 18:57:44.266864096 +0000 UTC m=+2167.498102378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls") pod "maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb" (UID: "b7dc1489-f35d-4533-a74a-796288531f74") : secret "maas-api-e2e-praxis-a-3932ef-serving-cert" not found
Oct 31 18:57:43.769127 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.769100    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-metrics-certs\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:43.826937 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:43.826866    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/b7dc1489-f35d-4533-a74a-796288531f74-kube-api-access-pbm6m\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:44.273994 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:44.273953    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:44.276928 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:44.276891    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:44.456808 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:44.456764    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:44.840936 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:44.840906    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"]
Oct 31 18:57:44.842676 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:57:44.842647    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7dc1489_f35d_4533_a74a_796288531f74.slice/crio-89b4402c31645782cb64c71d620e6ce6a7a0792f9c046c7657ddb54cd8034831 WatchSource:0}: Error finding container 89b4402c31645782cb64c71d620e6ce6a7a0792f9c046c7657ddb54cd8034831: Status 404 returned error can't find the container with id 89b4402c31645782cb64c71d620e6ce6a7a0792f9c046c7657ddb54cd8034831
Oct 31 18:57:45.172526 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:45.172490    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb" event={"ID":"b7dc1489-f35d-4533-a74a-796288531f74","Type":"ContainerStarted","Data":"89b4402c31645782cb64c71d620e6ce6a7a0792f9c046c7657ddb54cd8034831"}
Oct 31 18:57:46.200542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:46.198216    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb" event={"ID":"b7dc1489-f35d-4533-a74a-796288531f74","Type":"ContainerStarted","Data":"337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62"}
Oct 31 18:57:46.200542 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:46.199782    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:46.331429 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:46.331368    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb" podStartSLOduration=2.28835035 podStartE2EDuration="3.331351572s" totalImagesPullingTime="1.043001222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:57:46.328450226 +0000 UTC m=+2169.559688517" watchObservedRunningTime="2026-10-31 18:57:46.331351572 +0000 UTC m=+2169.562589906"
Oct 31 18:57:46.630649 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:57:46.630619    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:57:53.227502 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:53.227438    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 18:57:56.289095 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.289012    5551 generic.go:356] "Generic (PLEG): container finished" podID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerID="a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5" exitCode=0
Oct 31 18:57:56.289095 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.289080    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" event={"ID":"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35","Type":"ContainerDied","Data":"a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5"}
Oct 31 18:57:56.289759 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.289514    5551 scope.go:122] "RemoveContainer" containerID="a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5"
Oct 31 18:57:56.291127 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.291106    5551 generic.go:356] "Generic (PLEG): container finished" podID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerID="aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f" exitCode=0
Oct 31 18:57:56.291255 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.291137    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:57:56.291255 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.291181    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" event={"ID":"0a940ce0-f138-492c-84b6-aaf10d45a459","Type":"ContainerDied","Data":"aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f"}
Oct 31 18:57:56.291255 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.291214    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:57:56.291255 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.291248    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:57:56.291563 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.291545    5551 scope.go:122] "RemoveContainer" containerID="aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f"
Oct 31 18:57:56.293084 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.293055    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:57:56.293192 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.293135    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:57:56.293263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:56.293188    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:57:57.317515 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:57.317451    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" event={"ID":"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35","Type":"ContainerStarted","Data":"05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef"}
Oct 31 18:57:57.320313 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:57.320285    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" event={"ID":"0a940ce0-f138-492c-84b6-aaf10d45a459","Type":"ContainerStarted","Data":"0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f"}
Oct 31 18:57:57.619882 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:57.619783    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:57:57.621015 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:57.620988    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:57:57.621121 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:57.621038    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 31 18:57:57.723649 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:57.723607    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:57:57.724365 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:57.724339    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 18:57:57.724630 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:57.724590    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 18:57:58.619473 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:58.619422    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:57:58.619867 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:58.619508    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 31 18:57:58.724591 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:58.724550    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 18:57:58.724784 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:58.724639    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 18:57:59.619811 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:59.619776    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:57:59.620210 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:59.619840    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 31 18:57:59.723991 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:59.723962    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 18:57:59.724142 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:57:59.724020    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 18:58:00.538299 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:00.538264    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:58:00.619730 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:00.619691    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:58:00.619967 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:00.619936    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 31 18:58:00.724739 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:00.724684    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 18:58:00.724954 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:00.724934    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 18:58:01.620028 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:01.619991    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:58:01.620442 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:01.620044    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 31 18:58:01.724648 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:01.724617    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 18:58:01.724935 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:01.724911    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 18:58:02.615360 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:02.615324    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:58:02.740742 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:02.740708    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:58:02.765225 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:02.765191    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:58:02.771040 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:02.771006    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 18:58:03.409654 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:03.409628    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 18:58:04.186973 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.186938    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"]
Oct 31 18:58:04.190948 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.190926    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.247571 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.247547    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"]
Oct 31 18:58:04.281312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.281272    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.281312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.281318    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbwm\" (UniqueName: \"kubernetes.io/projected/28562d13-5ce8-41c9-b134-dcde44413880-kube-api-access-grbwm\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.281613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.281429    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28562d13-5ce8-41c9-b134-dcde44413880-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.281613 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.281505    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-metrics-certs\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.292717 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.292694    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-713bf0-serving-cert\""
Oct 31 18:58:04.383084 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.383044    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28562d13-5ce8-41c9-b134-dcde44413880-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.383272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.383094    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-metrics-certs\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.383272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.383236    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.383272 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.383261    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbwm\" (UniqueName: \"kubernetes.io/projected/28562d13-5ce8-41c9-b134-dcde44413880-kube-api-access-grbwm\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.383448 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:58:04.383391    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-serving-cert: secret "maas-api-e2e-ipp-a-713bf0-serving-cert" not found
Oct 31 18:58:04.383541 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:58:04.383495    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls podName:28562d13-5ce8-41c9-b134-dcde44413880 nodeName:}" failed. No retries permitted until 2026-10-31 18:58:04.883438736 +0000 UTC m=+2188.114677004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls") pod "maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6" (UID: "28562d13-5ce8-41c9-b134-dcde44413880") : secret "maas-api-e2e-ipp-a-713bf0-serving-cert" not found
Oct 31 18:58:04.383826 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.383805    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28562d13-5ce8-41c9-b134-dcde44413880-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.385847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.385822    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-metrics-certs\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.440786 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.440720    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbwm\" (UniqueName: \"kubernetes.io/projected/28562d13-5ce8-41c9-b134-dcde44413880-kube-api-access-grbwm\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.891119 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.891086    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:04.893687 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:04.893656    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls\") pod \"maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:05.106947 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:05.106910    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:05.481833 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:05.481805    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"]
Oct 31 18:58:05.483569 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:58:05.483538    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28562d13_5ce8_41c9_b134_dcde44413880.slice/crio-1c01473f75cc030e01fda5689ed700d258b27da7542e56ad0b48a59a1cb55ea3 WatchSource:0}: Error finding container 1c01473f75cc030e01fda5689ed700d258b27da7542e56ad0b48a59a1cb55ea3: Status 404 returned error can't find the container with id 1c01473f75cc030e01fda5689ed700d258b27da7542e56ad0b48a59a1cb55ea3
Oct 31 18:58:06.430186 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:06.430144    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6" event={"ID":"28562d13-5ce8-41c9-b134-dcde44413880","Type":"ContainerStarted","Data":"0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962"}
Oct 31 18:58:06.430344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:06.430211    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:06.430344 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:06.430236    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6" event={"ID":"28562d13-5ce8-41c9-b134-dcde44413880","Type":"ContainerStarted","Data":"1c01473f75cc030e01fda5689ed700d258b27da7542e56ad0b48a59a1cb55ea3"}
Oct 31 18:58:06.550628 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:06.550537    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6" podStartSLOduration=1.780928633 podStartE2EDuration="2.550522616s" totalImagesPullingTime="769.593983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:58:06.548281447 +0000 UTC m=+2189.779519738" watchObservedRunningTime="2026-10-31 18:58:06.550522616 +0000 UTC m=+2189.781761156"
Oct 31 18:58:12.442234 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:12.442206    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 18:58:16.604955 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:58:16.604911    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:58:27.677000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:27.627185    5551 generic.go:356] "Generic (PLEG): container finished" podID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerID="8de1d6332ee4dd0b1b1e2ff8b0cbcbcb04ea109681639b861e0f928c8218217e" exitCode=0
Oct 31 18:58:27.677000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:27.627263    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" event={"ID":"320c4562-4bfc-4db1-8839-068ae02a7dc9","Type":"ContainerDied","Data":"8de1d6332ee4dd0b1b1e2ff8b0cbcbcb04ea109681639b861e0f928c8218217e"}
Oct 31 18:58:27.677000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:27.627938    5551 scope.go:122] "RemoveContainer" containerID="8de1d6332ee4dd0b1b1e2ff8b0cbcbcb04ea109681639b861e0f928c8218217e"
Oct 31 18:58:27.677000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:27.630252    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:27.677000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:27.630362    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:27.677000 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:27.630405    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:27.713630 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:27.713597    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:58:27.713778 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:27.713674    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:58:28.636893 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.636856    5551 generic.go:356] "Generic (PLEG): container finished" podID="5ba30e08-8922-4c58-bf01-812abde01f55" containerID="aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e" exitCode=0
Oct 31 18:58:28.637083 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.636940    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" event={"ID":"5ba30e08-8922-4c58-bf01-812abde01f55","Type":"ContainerDied","Data":"aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e"}
Oct 31 18:58:28.637896 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.637416    5551 scope.go:122] "RemoveContainer" containerID="aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e"
Oct 31 18:58:28.639057 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.639027    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:28.639190 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.639117    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:28.639190 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.639168    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:28.639321 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.639301    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" event={"ID":"320c4562-4bfc-4db1-8839-068ae02a7dc9","Type":"ContainerStarted","Data":"f8e3a49d6ad6ec06fbc48c68d38570c87b5cd0f5f3904449673654ef4131fa7e"}
Oct 31 18:58:28.739204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.739177    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:58:28.740735 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.740711    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 18:58:28.740839 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:28.740768    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 18:58:29.647526 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:29.647483    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" event={"ID":"5ba30e08-8922-4c58-bf01-812abde01f55","Type":"ContainerStarted","Data":"c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e"}
Oct 31 18:58:29.739728 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:29.739695    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 18:58:29.740126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:29.739740    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 18:58:30.071539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:30.068574    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:58:30.071539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:30.068732    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 18:58:30.071539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:30.068776    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 18:58:30.740145 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:30.740111    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 18:58:30.740677 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:30.740165    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 18:58:30.801483 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:30.801163    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:58:31.081543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:31.067998    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 18:58:31.081543 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:31.068054    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 18:58:31.739606 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:31.739570    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 18:58:31.739797 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:31.739622    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 18:58:32.077541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:32.076871    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 18:58:32.077541 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:32.076938    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 18:58:32.739464 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:32.739430    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 18:58:32.739659 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:32.739507    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 18:58:33.068538 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:33.067760    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 18:58:33.068538 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:33.067819    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 18:58:33.739983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:33.739916    5551 patch_prober.go:29] interesting pod/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 18:58:33.740577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:33.740541    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 18:58:34.070539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:34.069744    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 18:58:34.070539 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:34.069800    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 18:58:34.744959 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:34.744927    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:58:34.746032 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:34.746010    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 18:58:35.086534 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:35.086402    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:58:35.086994 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:35.086965    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:58:35.087853 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:35.087833    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 18:58:46.604280 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:58:46.604242    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:58:58.310421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.310375    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"]
Oct 31 18:58:58.313974 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.313956    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.413493 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.411306    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b11e44-openshift-default-dockercfg-v7vcz\""
Oct 31 18:58:58.419263 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.419218    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"]
Oct 31 18:58:58.477416 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477369    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istiod-ca-cert\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.477638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477434    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-podinfo\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.477638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477569    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-data\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.477638 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477613    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-socket\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.477817 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477664    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-credential-socket\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.477817 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477729    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-certs\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.477817 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477770    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-token\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.477817 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477800    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-envoy\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.478009 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.477863    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j6cj\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-kube-api-access-8j6cj\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579063 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.578972    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j6cj\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-kube-api-access-8j6cj\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579238 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579065    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istiod-ca-cert\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579238 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579114    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-podinfo\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579238 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579206    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-data\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579248    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-socket\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579301    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-credential-socket\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579360    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-certs\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579414 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579400    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-token\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579639 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579430    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-envoy\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579914 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579717    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-socket\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.579914 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579869    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-data\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.580094 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.579930    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istiod-ca-cert\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.580094 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.580040    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-credential-socket\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.580218 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.580136    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-certs\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.581969 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.581942    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-podinfo\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.582108 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.582087    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-envoy\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.638094 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.638069    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-token\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.638558 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.638537    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j6cj\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-kube-api-access-8j6cj\") pod \"e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") " pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:58.927311 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:58.927280    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:58:59.366407 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:59.366378    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"]
Oct 31 18:58:59.368243 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:58:59.368214    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc92adf_66a4_4b82_8d77_d1d6760af6f3.slice/crio-742ef78ba0e8604659268dbfde7478d589c7047d54d70f828868be62b92b5e56 WatchSource:0}: Error finding container 742ef78ba0e8604659268dbfde7478d589c7047d54d70f828868be62b92b5e56: Status 404 returned error can't find the container with id 742ef78ba0e8604659268dbfde7478d589c7047d54d70f828868be62b92b5e56
Oct 31 18:58:59.371093 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:59.371050    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:59.371206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:59.371135    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:59.371206 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:59.371178    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 18:58:59.929421 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:59.929388    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" event={"ID":"bcc92adf-66a4-4b82-8d77-d1d6760af6f3","Type":"ContainerStarted","Data":"6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41"}
Oct 31 18:58:59.929620 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:58:59.929440    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" event={"ID":"bcc92adf-66a4-4b82-8d77-d1d6760af6f3","Type":"ContainerStarted","Data":"742ef78ba0e8604659268dbfde7478d589c7047d54d70f828868be62b92b5e56"}
Oct 31 18:59:00.099710 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:00.099652    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" podStartSLOduration=2.099630064 podStartE2EDuration="2.099630064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:59:00.094329303 +0000 UTC m=+2243.325567589" watchObservedRunningTime="2026-10-31 18:59:00.099630064 +0000 UTC m=+2243.330868358"
Oct 31 18:59:00.928297 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:00.928269    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:59:00.934082 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:00.934058    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:59:00.935558 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:00.935533    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:59:00.936252 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:00.936234    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 18:59:16.605902 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:59:16.605821    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:59:20.850097 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:20.850060    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"]
Oct 31 18:59:20.850597 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:20.850295    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k" podUID="486e62d9-fead-4f93-aaa6-cc565db5aaf4" containerName="maas-api" containerID="cri-o://c98a04645bfe82327b7b55bfeb3b32b602151e7066c7813554c182a837c5e964" gracePeriod=30
Oct 31 18:59:20.954632 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:20.954431    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"]
Oct 31 18:59:21.079077 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.079047    5551 generic.go:356] "Generic (PLEG): container finished" podID="486e62d9-fead-4f93-aaa6-cc565db5aaf4" containerID="c98a04645bfe82327b7b55bfeb3b32b602151e7066c7813554c182a837c5e964" exitCode=0
Oct 31 18:59:21.079251 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.079118    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k" event={"ID":"486e62d9-fead-4f93-aaa6-cc565db5aaf4","Type":"ContainerDied","Data":"c98a04645bfe82327b7b55bfeb3b32b602151e7066c7813554c182a837c5e964"}
Oct 31 18:59:21.111983 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.111924    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:59:21.268239 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.268210    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-maas-api-tls\") pod \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") "
Oct 31 18:59:21.268420 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.268257    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486e62d9-fead-4f93-aaa6-cc565db5aaf4-kube-api-access-mtlkh\" (UniqueName: \"kubernetes.io/projected/486e62d9-fead-4f93-aaa6-cc565db5aaf4-kube-api-access-mtlkh\") pod \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") "
Oct 31 18:59:21.268420 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.268369    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-metrics-certs\") pod \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") "
Oct 31 18:59:21.268587 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.268423    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486e62d9-fead-4f93-aaa6-cc565db5aaf4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486e62d9-fead-4f93-aaa6-cc565db5aaf4-openshift-service-ca\") pod \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\" (UID: \"486e62d9-fead-4f93-aaa6-cc565db5aaf4\") "
Oct 31 18:59:21.269088 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.269050    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486e62d9-fead-4f93-aaa6-cc565db5aaf4-openshift-service-ca" pod "486e62d9-fead-4f93-aaa6-cc565db5aaf4" (UID: "486e62d9-fead-4f93-aaa6-cc565db5aaf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:59:21.279498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.279072    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-maas-api-tls" pod "486e62d9-fead-4f93-aaa6-cc565db5aaf4" (UID: "486e62d9-fead-4f93-aaa6-cc565db5aaf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:59:21.279498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.279142    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486e62d9-fead-4f93-aaa6-cc565db5aaf4-kube-api-access-mtlkh" pod "486e62d9-fead-4f93-aaa6-cc565db5aaf4" (UID: "486e62d9-fead-4f93-aaa6-cc565db5aaf4"). InnerVolumeSpecName "kube-api-access-mtlkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:59:21.279498 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.279277    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-metrics-certs" pod "486e62d9-fead-4f93-aaa6-cc565db5aaf4" (UID: "486e62d9-fead-4f93-aaa6-cc565db5aaf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:59:21.370158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.370081    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:21.370158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.370111    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlkh\" (UniqueName: \"kubernetes.io/projected/486e62d9-fead-4f93-aaa6-cc565db5aaf4-kube-api-access-mtlkh\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:21.370158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.370122    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/486e62d9-fead-4f93-aaa6-cc565db5aaf4-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:21.370158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:21.370130    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486e62d9-fead-4f93-aaa6-cc565db5aaf4-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:22.087174 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:22.087143    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"
Oct 31 18:59:22.087669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:22.087193    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k" event={"ID":"486e62d9-fead-4f93-aaa6-cc565db5aaf4","Type":"ContainerDied","Data":"2452a6f65fcec2a7dcb9234d45e5feaec2458ce6c8e4cb0df2f225b2bb86c1b4"}
Oct 31 18:59:22.087669 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:22.087258    5551 scope.go:122] "RemoveContainer" containerID="c98a04645bfe82327b7b55bfeb3b32b602151e7066c7813554c182a837c5e964"
Oct 31 18:59:22.251738 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:22.251659    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"]
Oct 31 18:59:22.318023 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:22.317990    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84c46e-ffdd75775-zcs7k"]
Oct 31 18:59:22.910389 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:22.910355    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486e62d9-fead-4f93-aaa6-cc565db5aaf4" path="/var/lib/kubelet/pods/486e62d9-fead-4f93-aaa6-cc565db5aaf4/volumes"
Oct 31 18:59:24.710244 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:24.710208    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"]
Oct 31 18:59:24.710732 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:24.710646    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy" containerID="cri-o://835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841" gracePeriod=30
Oct 31 18:59:25.955079 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:25.955042    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk" podUID="4ef3112e-e1cd-4899-8af7-d2188fcb2e40" containerName="payload-pre-processing" containerID="cri-o://3d429f0cfbfbcba82f5b0088b89773d1bf72a73e697276d611a46e73e63a79f8" gracePeriod=25
Oct 31 18:59:26.120392 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.120355    5551 generic.go:356] "Generic (PLEG): container finished" podID="4ef3112e-e1cd-4899-8af7-d2188fcb2e40" containerID="3d429f0cfbfbcba82f5b0088b89773d1bf72a73e697276d611a46e73e63a79f8" exitCode=0
Oct 31 18:59:26.120577 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.120427    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk" event={"ID":"4ef3112e-e1cd-4899-8af7-d2188fcb2e40","Type":"ContainerDied","Data":"3d429f0cfbfbcba82f5b0088b89773d1bf72a73e697276d611a46e73e63a79f8"}
Oct 31 18:59:26.228905 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.228871    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:59:26.246322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.246291    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-kube-api-access-gl9db\" (UniqueName: \"kubernetes.io/projected/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-kube-api-access-gl9db\") pod \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\" (UID: \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\") "
Oct 31 18:59:26.247014 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.246585    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-plugins-config-volume\") pod \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\" (UID: \"4ef3112e-e1cd-4899-8af7-d2188fcb2e40\") "
Oct 31 18:59:26.247097 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.247033    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-plugins-config-volume" pod "4ef3112e-e1cd-4899-8af7-d2188fcb2e40" (UID: "4ef3112e-e1cd-4899-8af7-d2188fcb2e40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:59:26.247327 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.247306    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:26.248650 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.248624    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-kube-api-access-gl9db" pod "4ef3112e-e1cd-4899-8af7-d2188fcb2e40" (UID: "4ef3112e-e1cd-4899-8af7-d2188fcb2e40"). InnerVolumeSpecName "kube-api-access-gl9db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:59:26.348269 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:26.348233    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl9db\" (UniqueName: \"kubernetes.io/projected/4ef3112e-e1cd-4899-8af7-d2188fcb2e40-kube-api-access-gl9db\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:27.129215 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:27.129184    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"
Oct 31 18:59:27.129713 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:27.129230    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk" event={"ID":"4ef3112e-e1cd-4899-8af7-d2188fcb2e40","Type":"ContainerDied","Data":"44849b5cac60c0db1389cfd87e987c2b3d4fa4c046c8cb99a90f3c1c99ab2f06"}
Oct 31 18:59:27.129713 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:27.129305    5551 scope.go:122] "RemoveContainer" containerID="3d429f0cfbfbcba82f5b0088b89773d1bf72a73e697276d611a46e73e63a79f8"
Oct 31 18:59:27.293634 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:27.293546    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"]
Oct 31 18:59:27.343351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:27.343321    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-68695c8fc5-hs2zk"]
Oct 31 18:59:28.910388 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:28.910353    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef3112e-e1cd-4899-8af7-d2188fcb2e40" path="/var/lib/kubelet/pods/4ef3112e-e1cd-4899-8af7-d2188fcb2e40/volumes"
Oct 31 18:59:29.989912 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:29.989839    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:59:30.100809 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.100768    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-podinfo\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.100828    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-data\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.100860    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-credential-socket\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101005 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.100937    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-token\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-token\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101013    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istiod-ca-cert\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101072    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-envoy\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101108    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-socket\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101165 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101153    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-certs\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101178    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-data" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:59:30.101329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101192    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-credential-socket" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:59:30.101329 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101194    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-kube-api-access-c824h\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-kube-api-access-c824h\") pod \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\" (UID: \"7999f6f9-a7e8-4ff7-a9de-57aefb668280\") "
Oct 31 18:59:30.101510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101375    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istiod-ca-cert" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:59:30.101510 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101393    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-socket" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:59:30.101932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101817    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.101932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101843    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.101932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101859    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.101932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101870    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.101932 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.101842    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-certs" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:59:30.103316 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.103289    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-podinfo" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:59:30.103507 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.103483    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-kube-api-access-c824h" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "kube-api-access-c824h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:59:30.103589 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.103565    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-token" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:59:30.103656 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.103643    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-envoy" pod "7999f6f9-a7e8-4ff7-a9de-57aefb668280" (UID: "7999f6f9-a7e8-4ff7-a9de-57aefb668280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:59:30.153179 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.153140    5551 generic.go:356] "Generic (PLEG): container finished" podID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerID="835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841" exitCode=0
Oct 31 18:59:30.153393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.153200    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"
Oct 31 18:59:30.153393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.153227    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" event={"ID":"7999f6f9-a7e8-4ff7-a9de-57aefb668280","Type":"ContainerDied","Data":"835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841"}
Oct 31 18:59:30.153393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.153345    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc" event={"ID":"7999f6f9-a7e8-4ff7-a9de-57aefb668280","Type":"ContainerDied","Data":"aa196a29b865aacc4a1db6b93429cf80fca62b11a0de36a14f8d00e781366b64"}
Oct 31 18:59:30.153393 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.153376    5551 scope.go:122] "RemoveContainer" containerID="835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841"
Oct 31 18:59:30.167196 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.167172    5551 scope.go:122] "RemoveContainer" containerID="484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06"
Oct 31 18:59:30.177089 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.177066    5551 scope.go:122] "RemoveContainer" containerID="835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841"
Oct 31 18:59:30.177398 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:59:30.177380    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841\": container with ID starting with 835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841 not found: ID does not exist" containerID="835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841"
Oct 31 18:59:30.177446 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.177409    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841"} err="failed to get container status \"835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841\": rpc error: code = NotFound desc = could not find container \"835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841\": container with ID starting with 835657ca748b177b0142e965c6fa2d2fb6de60902ea8707c99c4b0036b06d841 not found: ID does not exist"
Oct 31 18:59:30.177446 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.177426    5551 scope.go:122] "RemoveContainer" containerID="484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06"
Oct 31 18:59:30.177721 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:59:30.177702    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06\": container with ID starting with 484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06 not found: ID does not exist" containerID="484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06"
Oct 31 18:59:30.177778 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.177727    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06"} err="failed to get container status \"484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06\": rpc error: code = NotFound desc = could not find container \"484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06\": container with ID starting with 484108e86ca0e92b4e107f72110c50b0049e12468f4a58994a9922d7101fbe06 not found: ID does not exist"
Oct 31 18:59:30.204159 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.204120    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.204312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.204164    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.204312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.204181    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7999f6f9-a7e8-4ff7-a9de-57aefb668280-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.204312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.204195    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c824h\" (UniqueName: \"kubernetes.io/projected/7999f6f9-a7e8-4ff7-a9de-57aefb668280-kube-api-access-c824h\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.204312 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.204211    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7999f6f9-a7e8-4ff7-a9de-57aefb668280-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:30.321136 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.321056    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"]
Oct 31 18:59:30.327007 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.326985    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-84c46e-openshift-default-64cb75d877-jh7tc"]
Oct 31 18:59:30.909351 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:30.909311    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" path="/var/lib/kubelet/pods/7999f6f9-a7e8-4ff7-a9de-57aefb668280/volumes"
Oct 31 18:59:31.866322 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:31.866289    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"]
Oct 31 18:59:31.866860 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:31.866528    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs" podUID="df2fc59f-cd6e-4d4c-a0a4-901e79375b37" containerName="maas-api" containerID="cri-o://6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885" gracePeriod=30
Oct 31 18:59:32.146564 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.146512    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:59:32.172938 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.172902    5551 generic.go:356] "Generic (PLEG): container finished" podID="df2fc59f-cd6e-4d4c-a0a4-901e79375b37" containerID="6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885" exitCode=0
Oct 31 18:59:32.173155 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.172955    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"
Oct 31 18:59:32.173155 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.172955    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs" event={"ID":"df2fc59f-cd6e-4d4c-a0a4-901e79375b37","Type":"ContainerDied","Data":"6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885"}
Oct 31 18:59:32.173155 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.173025    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs" event={"ID":"df2fc59f-cd6e-4d4c-a0a4-901e79375b37","Type":"ContainerDied","Data":"f567a85f93bc5673d3e92493e9376f17bd6007c86849177b7b4d40d2efd40d65"}
Oct 31 18:59:32.173155 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.173056    5551 scope.go:122] "RemoveContainer" containerID="6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885"
Oct 31 18:59:32.186805 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.186452    5551 scope.go:122] "RemoveContainer" containerID="6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885"
Oct 31 18:59:32.187158 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:59:32.187058    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885\": container with ID starting with 6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885 not found: ID does not exist" containerID="6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885"
Oct 31 18:59:32.187158 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.187099    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885"} err="failed to get container status \"6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885\": rpc error: code = NotFound desc = could not find container \"6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885\": container with ID starting with 6402ee8bf78590f994c0d7f11f6136e70100a750bcaec92d8a682283cbd57885 not found: ID does not exist"
Oct 31 18:59:32.225946 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.225908    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-openshift-service-ca\") pod \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") "
Oct 31 18:59:32.226663 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.226640    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-kube-api-access-mwz7v\") pod \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") "
Oct 31 18:59:32.226856 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.226832    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-metrics-certs\") pod \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") "
Oct 31 18:59:32.227006 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.226983    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-maas-api-tls\") pod \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\" (UID: \"df2fc59f-cd6e-4d4c-a0a4-901e79375b37\") "
Oct 31 18:59:32.237527 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.228880    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-openshift-service-ca" pod "df2fc59f-cd6e-4d4c-a0a4-901e79375b37" (UID: "df2fc59f-cd6e-4d4c-a0a4-901e79375b37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:59:32.269126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.266158    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-metrics-certs" pod "df2fc59f-cd6e-4d4c-a0a4-901e79375b37" (UID: "df2fc59f-cd6e-4d4c-a0a4-901e79375b37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:59:32.269126 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.267903    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-maas-api-tls" pod "df2fc59f-cd6e-4d4c-a0a4-901e79375b37" (UID: "df2fc59f-cd6e-4d4c-a0a4-901e79375b37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:59:32.273508 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.269625    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-kube-api-access-mwz7v" pod "df2fc59f-cd6e-4d4c-a0a4-901e79375b37" (UID: "df2fc59f-cd6e-4d4c-a0a4-901e79375b37"). InnerVolumeSpecName "kube-api-access-mwz7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:59:32.329099 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.329070    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-kube-api-access-mwz7v\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:32.329099 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.329099    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:32.329296 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.329109    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:32.329296 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.329118    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fc59f-cd6e-4d4c-a0a4-901e79375b37-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:32.619534 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.619495    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"]
Oct 31 18:59:32.623248 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.623224    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-35566a-68bb9c79f6-dtzxs"]
Oct 31 18:59:32.951492 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:32.941037    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2fc59f-cd6e-4d4c-a0a4-901e79375b37" path="/var/lib/kubelet/pods/df2fc59f-cd6e-4d4c-a0a4-901e79375b37/volumes"
Oct 31 18:59:46.605028 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:59:46.604984    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 18:59:49.634847 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.634813    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"]
Oct 31 18:59:49.635394 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635378    5551 cpu_manager.go:414] "containerMap: removing container" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy"
Oct 31 18:59:49.635485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635396    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy"
Oct 31 18:59:49.635485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635411    5551 cpu_manager.go:414] "containerMap: removing container" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy"
Oct 31 18:59:49.635485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635416    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy"
Oct 31 18:59:49.635485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635435    5551 cpu_manager.go:414] "containerMap: removing container" podUID="df2fc59f-cd6e-4d4c-a0a4-901e79375b37" containerName="maas-api"
Oct 31 18:59:49.635485 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635446    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2fc59f-cd6e-4d4c-a0a4-901e79375b37" containerName="maas-api"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635489    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4ef3112e-e1cd-4899-8af7-d2188fcb2e40" containerName="payload-pre-processing"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635495    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef3112e-e1cd-4899-8af7-d2188fcb2e40" containerName="payload-pre-processing"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635506    5551 cpu_manager.go:414] "containerMap: removing container" podUID="486e62d9-fead-4f93-aaa6-cc565db5aaf4" containerName="maas-api"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635510    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486e62d9-fead-4f93-aaa6-cc565db5aaf4" containerName="maas-api"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635602    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef3112e-e1cd-4899-8af7-d2188fcb2e40" containerName="payload-pre-processing"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635611    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="486e62d9-fead-4f93-aaa6-cc565db5aaf4" containerName="maas-api"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635621    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635628    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="7999f6f9-a7e8-4ff7-a9de-57aefb668280" containerName="istio-proxy"
Oct 31 18:59:49.635665 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.635639    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2fc59f-cd6e-4d4c-a0a4-901e79375b37" containerName="maas-api"
Oct 31 18:59:49.640628 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.640608    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.693140 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.693104    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e43ccf-serving-cert\""
Oct 31 18:59:49.695113 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.695085    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"]
Oct 31 18:59:49.742055 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.742030    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhq8\" (UniqueName: \"kubernetes.io/projected/fe4cf1cb-a637-47b0-81c8-0478c86e5969-kube-api-access-zqhq8\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.742187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.742070    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-metrics-certs\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.742187 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.742089    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.742282 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.742242    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe4cf1cb-a637-47b0-81c8-0478c86e5969-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.842922 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.842887    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhq8\" (UniqueName: \"kubernetes.io/projected/fe4cf1cb-a637-47b0-81c8-0478c86e5969-kube-api-access-zqhq8\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.843081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.842932    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-metrics-certs\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.843081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.842950    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.843081 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.843000    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe4cf1cb-a637-47b0-81c8-0478c86e5969-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.843198 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:59:49.843100    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-serving-cert: secret "maas-api-e2e-praxis-b-e43ccf-serving-cert" not found
Oct 31 18:59:49.843198 ip-10-0-132-99 kubenswrapper[5551]: E1031 18:59:49.843174    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls podName:fe4cf1cb-a637-47b0-81c8-0478c86e5969 nodeName:}" failed. No retries permitted until 2026-10-31 18:59:50.343154823 +0000 UTC m=+2293.574393105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls") pod "maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x" (UID: "fe4cf1cb-a637-47b0-81c8-0478c86e5969") : secret "maas-api-e2e-praxis-b-e43ccf-serving-cert" not found
Oct 31 18:59:49.843664 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.843644    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe4cf1cb-a637-47b0-81c8-0478c86e5969-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.845508 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.845490    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-metrics-certs\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:49.902275 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:49.902207    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhq8\" (UniqueName: \"kubernetes.io/projected/fe4cf1cb-a637-47b0-81c8-0478c86e5969-kube-api-access-zqhq8\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:50.348723 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:50.348649    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:50.351318 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:50.351286    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:50.554052 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:50.554018    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:50.931204 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:50.931128    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"]
Oct 31 18:59:50.933137 ip-10-0-132-99 kubenswrapper[5551]: W1031 18:59:50.933112    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4cf1cb_a637_47b0_81c8_0478c86e5969.slice/crio-ad713b26e69f59f42c56f6b772aa169e1422e1f268347ed9ed9639dbefc3b39b WatchSource:0}: Error finding container ad713b26e69f59f42c56f6b772aa169e1422e1f268347ed9ed9639dbefc3b39b: Status 404 returned error can't find the container with id ad713b26e69f59f42c56f6b772aa169e1422e1f268347ed9ed9639dbefc3b39b
Oct 31 18:59:50.934658 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:50.934639    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:59:51.313962 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:51.313867    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x" event={"ID":"fe4cf1cb-a637-47b0-81c8-0478c86e5969","Type":"ContainerStarted","Data":"ad713b26e69f59f42c56f6b772aa169e1422e1f268347ed9ed9639dbefc3b39b"}
Oct 31 18:59:52.133240 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:52.133210    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"]
Oct 31 18:59:52.322364 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:52.322264    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x" event={"ID":"fe4cf1cb-a637-47b0-81c8-0478c86e5969","Type":"ContainerStarted","Data":"c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f"}
Oct 31 18:59:52.322364 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:52.322349    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 18:59:52.441888 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:52.441841    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x" podStartSLOduration=2.3957167950000002 podStartE2EDuration="3.441827081s" totalImagesPullingTime="1.046110286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:59:52.438875117 +0000 UTC m=+2295.670113407" watchObservedRunningTime="2026-10-31 18:59:52.441827081 +0000 UTC m=+2295.673065371"
Oct 31 18:59:58.334821 ip-10-0-132-99 kubenswrapper[5551]: I1031 18:59:58.334791    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 19:00:07.154518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.154433    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7" podUID="0eb97b27-3e63-4ef2-bad2-faebae4ac636" containerName="main" containerID="cri-o://eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813" gracePeriod=45
Oct 31 19:00:07.428279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.428250    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 19:00:07.432017 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.431990    5551 generic.go:356] "Generic (PLEG): container finished" podID="0eb97b27-3e63-4ef2-bad2-faebae4ac636" containerID="eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813" exitCode=0
Oct 31 19:00:07.432157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.432031    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"
Oct 31 19:00:07.432157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.432040    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7" event={"ID":"0eb97b27-3e63-4ef2-bad2-faebae4ac636","Type":"ContainerDied","Data":"eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813"}
Oct 31 19:00:07.432157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.432098    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7" event={"ID":"0eb97b27-3e63-4ef2-bad2-faebae4ac636","Type":"ContainerDied","Data":"48aeb0149b4668ed78f6e2bf6537fbb4b49f40700e5d19dff5cd348c56677f93"}
Oct 31 19:00:07.432157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.432123    5551 scope.go:122] "RemoveContainer" containerID="eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813"
Oct 31 19:00:07.442811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.442790    5551 scope.go:122] "RemoveContainer" containerID="eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813"
Oct 31 19:00:07.443080 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:00:07.443060    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813\": container with ID starting with eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813 not found: ID does not exist" containerID="eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813"
Oct 31 19:00:07.443129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.443093    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813"} err="failed to get container status \"eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813\": rpc error: code = NotFound desc = could not find container \"eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813\": container with ID starting with eb8704314619e8f02e39c6775d4c9409242d807e159ef492edfbc36b8dc29813 not found: ID does not exist"
Oct 31 19:00:07.573393 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.573356    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb97b27-3e63-4ef2-bad2-faebae4ac636-kube-api-access-gkrx7\" (UniqueName: \"kubernetes.io/projected/0eb97b27-3e63-4ef2-bad2-faebae4ac636-kube-api-access-gkrx7\") pod \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") "
Oct 31 19:00:07.573592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.573425    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tls-certs\") pod \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") "
Oct 31 19:00:07.573592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.573489    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-home\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-home\") pod \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") "
Oct 31 19:00:07.573592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.573543    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-dshm\") pod \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") "
Oct 31 19:00:07.573721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.573618    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tmp-dir\") pod \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") "
Oct 31 19:00:07.573721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.573661    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-model-cache\") pod \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\" (UID: \"0eb97b27-3e63-4ef2-bad2-faebae4ac636\") "
Oct 31 19:00:07.574780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.574535    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-home" pod "0eb97b27-3e63-4ef2-bad2-faebae4ac636" (UID: "0eb97b27-3e63-4ef2-bad2-faebae4ac636"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:07.575004 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.574974    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tmp-dir" pod "0eb97b27-3e63-4ef2-bad2-faebae4ac636" (UID: "0eb97b27-3e63-4ef2-bad2-faebae4ac636"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:07.575118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.575097    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-model-cache" pod "0eb97b27-3e63-4ef2-bad2-faebae4ac636" (UID: "0eb97b27-3e63-4ef2-bad2-faebae4ac636"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:07.578416 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.578376    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-dshm" pod "0eb97b27-3e63-4ef2-bad2-faebae4ac636" (UID: "0eb97b27-3e63-4ef2-bad2-faebae4ac636"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:07.578664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.578635    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb97b27-3e63-4ef2-bad2-faebae4ac636-kube-api-access-gkrx7" pod "0eb97b27-3e63-4ef2-bad2-faebae4ac636" (UID: "0eb97b27-3e63-4ef2-bad2-faebae4ac636"). InnerVolumeSpecName "kube-api-access-gkrx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:07.586265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.586237    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tls-certs" pod "0eb97b27-3e63-4ef2-bad2-faebae4ac636" (UID: "0eb97b27-3e63-4ef2-bad2-faebae4ac636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:00:07.675935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.675842    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:07.675935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.675872    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:07.675935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.675882    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkrx7\" (UniqueName: \"kubernetes.io/projected/0eb97b27-3e63-4ef2-bad2-faebae4ac636-kube-api-access-gkrx7\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:07.675935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.675892    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb97b27-3e63-4ef2-bad2-faebae4ac636-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:07.675935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.675901    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:07.675935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.675908    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb97b27-3e63-4ef2-bad2-faebae4ac636-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:07.853139 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.853100    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"]
Oct 31 19:00:07.855685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:07.855665    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-676224/ipp-route-676224-kserve-699b8c644f-dvrv7"]
Oct 31 19:00:08.909125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:08.909093    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb97b27-3e63-4ef2-bad2-faebae4ac636" path="/var/lib/kubelet/pods/0eb97b27-3e63-4ef2-bad2-faebae4ac636/volumes"
Oct 31 19:00:10.310764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.310732    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"]
Oct 31 19:00:10.311343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.311328    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0eb97b27-3e63-4ef2-bad2-faebae4ac636" containerName="main"
Oct 31 19:00:10.311397 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.311345    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb97b27-3e63-4ef2-bad2-faebae4ac636" containerName="main"
Oct 31 19:00:10.311449 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.311440    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb97b27-3e63-4ef2-bad2-faebae4ac636" containerName="main"
Oct 31 19:00:10.316161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.316144    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.371319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.371296    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"]
Oct 31 19:00:10.416370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.416336    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b11e44-serving-cert\""
Oct 31 19:00:10.510012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.509976    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjff8\" (UniqueName: \"kubernetes.io/projected/adff5c77-695a-4300-af20-ee81d7fa13d7-kube-api-access-wjff8\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.510159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.510025    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-metrics-certs\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.510159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.510138    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.510311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.510292    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adff5c77-695a-4300-af20-ee81d7fa13d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.611754 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.611670    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjff8\" (UniqueName: \"kubernetes.io/projected/adff5c77-695a-4300-af20-ee81d7fa13d7-kube-api-access-wjff8\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.611754 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.611731    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-metrics-certs\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.611982 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.611814    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.611982 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.611967    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adff5c77-695a-4300-af20-ee81d7fa13d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.612554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.612528    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adff5c77-695a-4300-af20-ee81d7fa13d7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.614324 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.614299    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.614429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.614412    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-metrics-certs\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.667772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.667731    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjff8\" (UniqueName: \"kubernetes.io/projected/adff5c77-695a-4300-af20-ee81d7fa13d7-kube-api-access-wjff8\") pod \"maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:10.929378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:10.929337    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:11.314677 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:11.314650    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"]
Oct 31 19:00:11.317514 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:00:11.317483    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadff5c77_695a_4300_af20_ee81d7fa13d7.slice/crio-bd88273bcf2fba0814061001d49c427e4530afbd4ddbbeb83aef11dbc330c2ef WatchSource:0}: Error finding container bd88273bcf2fba0814061001d49c427e4530afbd4ddbbeb83aef11dbc330c2ef: Status 404 returned error can't find the container with id bd88273bcf2fba0814061001d49c427e4530afbd4ddbbeb83aef11dbc330c2ef
Oct 31 19:00:11.463304 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:11.463269    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n" event={"ID":"adff5c77-695a-4300-af20-ee81d7fa13d7","Type":"ContainerStarted","Data":"bd88273bcf2fba0814061001d49c427e4530afbd4ddbbeb83aef11dbc330c2ef"}
Oct 31 19:00:12.474197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:12.474161    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n" event={"ID":"adff5c77-695a-4300-af20-ee81d7fa13d7","Type":"ContainerStarted","Data":"8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02"}
Oct 31 19:00:12.474765 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:12.474348    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:12.589920 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:12.589855    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n" podStartSLOduration=1.809729251 podStartE2EDuration="2.589840517s" totalImagesPullingTime="780.111266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:12.586646211 +0000 UTC m=+2315.817884501" watchObservedRunningTime="2026-10-31 19:00:12.589840517 +0000 UTC m=+2315.821078806"
Oct 31 19:00:16.606642 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:00:16.606600    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:00:18.487936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:18.487902    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:00:39.873582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:39.873544    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"]
Oct 31 19:00:39.880265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:39.880234    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:39.977523 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:39.977489    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"]
Oct 31 19:00:39.990820 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:39.990787    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4c58c5-openshift-default-dockercfg-54lm7\""
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.013720    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762f8ee8-681f-495c-b902-75a8d1214edf-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.013781    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762f8ee8-681f-495c-b902-75a8d1214edf-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.013907    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.013972    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.014045    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.014137    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6mpv\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-kube-api-access-l6mpv\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.014260    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.014293    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.015307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.014365    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.115412 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115368    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.115671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115573    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762f8ee8-681f-495c-b902-75a8d1214edf-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.115671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115617    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762f8ee8-681f-495c-b902-75a8d1214edf-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.115671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115656    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.115848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115684    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.115848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115732    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.115848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115790    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6mpv\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-kube-api-access-l6mpv\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.116009 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115875    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.116009 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.115907    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.116389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.116361    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.118646 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.117969    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762f8ee8-681f-495c-b902-75a8d1214edf-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.120416 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.120104    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.120416 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.120369    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.127527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.127472    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762f8ee8-681f-495c-b902-75a8d1214edf-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.128044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.127981    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.132533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.132486    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.173909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.173861    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.176098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.176069    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6mpv\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-kube-api-access-l6mpv\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.195788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.195756    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:40.641410 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.637322    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"]
Oct 31 19:00:40.641506 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:00:40.641447    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762f8ee8_681f_495c_b902_75a8d1214edf.slice/crio-f12701d6dbeff98717dfcf948a45cf10dea3456609ee834fedbe853e5eb86928 WatchSource:0}: Error finding container f12701d6dbeff98717dfcf948a45cf10dea3456609ee834fedbe853e5eb86928: Status 404 returned error can't find the container with id f12701d6dbeff98717dfcf948a45cf10dea3456609ee834fedbe853e5eb86928
Oct 31 19:00:40.644540 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.644498    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:00:40.644642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.644594    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:00:40.644689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.644642    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:00:40.708915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:40.708889    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" event={"ID":"762f8ee8-681f-495c-b902-75a8d1214edf","Type":"ContainerStarted","Data":"f12701d6dbeff98717dfcf948a45cf10dea3456609ee834fedbe853e5eb86928"}
Oct 31 19:00:41.326386 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:41.326354    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"]
Oct 31 19:00:41.326852 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:41.326583    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy" containerID="cri-o://ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3" gracePeriod=30
Oct 31 19:00:41.718674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:41.718633    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" event={"ID":"762f8ee8-681f-495c-b902-75a8d1214edf","Type":"ContainerStarted","Data":"38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470"}
Oct 31 19:00:41.834928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:41.834864    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podStartSLOduration=2.834846745 podStartE2EDuration="2.834846745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:41.83206124 +0000 UTC m=+2345.063299531" watchObservedRunningTime="2026-10-31 19:00:41.834846745 +0000 UTC m=+2345.066085034"
Oct 31 19:00:42.196472 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.196431    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:42.202286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.202259    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:42.726589 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.726515    5551 generic.go:356] "Generic (PLEG): container finished" podID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerID="6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41" exitCode=0
Oct 31 19:00:42.726940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.726593    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" event={"ID":"bcc92adf-66a4-4b82-8d77-d1d6760af6f3","Type":"ContainerDied","Data":"6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41"}
Oct 31 19:00:42.727023 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.726991    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:42.727144 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.727126    5551 scope.go:122] "RemoveContainer" containerID="6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41"
Oct 31 19:00:42.728011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.727793    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:00:42.728644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.728621    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:00:42.728715 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.728684    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:00:42.728757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:42.728715    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:00:43.735266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:43.735227    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" event={"ID":"bcc92adf-66a4-4b82-8d77-d1d6760af6f3","Type":"ContainerStarted","Data":"c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208"}
Oct 31 19:00:43.927791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:43.927755    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 19:00:43.928977 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:43.928953    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:00:43.929059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:43.929003    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 31 19:00:44.928386 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:44.928352    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:00:44.928784 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:44.928416    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 31 19:00:45.183544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:45.182766    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"]
Oct 31 19:00:45.183544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:45.182991    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy" containerID="cri-o://05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef" gracePeriod=30
Oct 31 19:00:45.928434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:45.928399    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:00:45.928840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:45.928480    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 31 19:00:45.936545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:45.936521    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 19:00:46.631304 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:00:46.630660    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:00:46.696599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.696259    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 19:00:46.802431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.802392    5551 generic.go:356] "Generic (PLEG): container finished" podID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerID="ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3" exitCode=0
Oct 31 19:00:46.802431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.802423    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"
Oct 31 19:00:46.802723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.802576    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" event={"ID":"1ce15580-1369-48c7-b20e-cec1a82cb6ed","Type":"ContainerDied","Data":"ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3"}
Oct 31 19:00:46.802723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.802629    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" event={"ID":"1ce15580-1369-48c7-b20e-cec1a82cb6ed","Type":"ContainerDied","Data":"b9fde9fc0c4a374236b26c2eb8ecfa6bcd95520227b6422e9b8f9e2d0ac126d2"}
Oct 31 19:00:46.802723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.802676    5551 scope.go:122] "RemoveContainer" containerID="ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3"
Oct 31 19:00:46.813620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.813602    5551 scope.go:122] "RemoveContainer" containerID="7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7"
Oct 31 19:00:46.820971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.820948    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-certs\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821082 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821002    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-kube-api-access-qfzmn\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-kube-api-access-qfzmn\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821082 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821055    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-data\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821152    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-credential-socket\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821184    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-certs" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:46.821311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821205    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-podinfo\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821247    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-envoy\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821280    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-token\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-token\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821347    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-socket\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821350    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-data" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:46.821487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821377    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istiod-ca-cert\") pod \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\" (UID: \"1ce15580-1369-48c7-b20e-cec1a82cb6ed\") "
Oct 31 19:00:46.821644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821589    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-credential-socket" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:46.821726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821706    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-socket" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:46.821927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821902    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istiod-ca-cert" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:46.822016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821968    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:46.822016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821986    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:46.822016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.821996    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:46.822016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.822005    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:46.824049 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.823872    5551 scope.go:122] "RemoveContainer" containerID="ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3"
Oct 31 19:00:46.824221 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:00:46.824202    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3\": container with ID starting with ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3 not found: ID does not exist" containerID="ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3"
Oct 31 19:00:46.824291 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.824235    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3"} err="failed to get container status \"ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3\": rpc error: code = NotFound desc = could not find container \"ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3\": container with ID starting with ba4fbcdac13818ea39bfe2cca748808393477c8e34aa4b1ab21804e3ec5208a3 not found: ID does not exist"
Oct 31 19:00:46.824291 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.824255    5551 scope.go:122] "RemoveContainer" containerID="7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7"
Oct 31 19:00:46.824584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.824552    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-podinfo" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:00:46.824584 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:00:46.824561    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7\": container with ID starting with 7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7 not found: ID does not exist" containerID="7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7"
Oct 31 19:00:46.824706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.824594    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7"} err="failed to get container status \"7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7\": rpc error: code = NotFound desc = could not find container \"7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7\": container with ID starting with 7b97382356a5d11e97423796c49ca21ec1b1cfbbaf659f2019b08181dda2d8f7 not found: ID does not exist"
Oct 31 19:00:46.824706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.824571    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-kube-api-access-qfzmn" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "kube-api-access-qfzmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:46.824706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.824635    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-token" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:46.824803 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.824771    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-envoy" pod "1ce15580-1369-48c7-b20e-cec1a82cb6ed" (UID: "1ce15580-1369-48c7-b20e-cec1a82cb6ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:46.929584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.928943    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:00:46.929584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.929041    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 31 19:00:46.930074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.929790    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:46.930074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.929807    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:46.930074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.929819    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:46.930074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.929828    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzmn\" (UniqueName: \"kubernetes.io/projected/1ce15580-1369-48c7-b20e-cec1a82cb6ed-kube-api-access-qfzmn\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:46.930074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:46.929841    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce15580-1369-48c7-b20e-cec1a82cb6ed-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:47.223219 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:47.223130    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"]
Oct 31 19:00:47.226777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:47.226751    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7"]
Oct 31 19:00:47.569554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:47.565550    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:00:47.569554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:47.565638    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-qnzj7" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:00:47.783418 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:47.783370    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:00:47.783748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:47.783704    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:00:47.928116 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:47.928081    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:00:47.928293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:47.928138    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 31 19:00:48.912059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:48.912024    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" path="/var/lib/kubelet/pods/1ce15580-1369-48c7-b20e-cec1a82cb6ed/volumes"
Oct 31 19:00:49.127179 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:49.127135    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 19:00:49.128817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:49.128788    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 19:00:50.453919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.453858    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 19:00:50.516147 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516113    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istiod-ca-cert\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516195    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-envoy\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516290    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-certs\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516341    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-token\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-token\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516379    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-credential-socket\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516577 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516429    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-podinfo\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516577 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516490    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-data\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516577 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516538    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istiod-ca-cert" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:50.516577 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516545    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-socket\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516676    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-certs" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:50.516778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516697    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-credential-socket" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:50.516778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516720    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-kube-api-access-v4jhp\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-kube-api-access-v4jhp\") pod \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\" (UID: \"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35\") "
Oct 31 19:00:50.516941 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.516839    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-data" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:50.517212 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.517188    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-socket" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:50.517636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.517563    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.517636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.517595    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.517636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.517612    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.517859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.517629    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.517859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.517681    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.518765 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.518741    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-token" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:50.518868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.518790    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-podinfo" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:00:50.518868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.518808    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-envoy" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:00:50.519183 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.519169    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-kube-api-access-v4jhp" pod "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" (UID: "8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35"). InnerVolumeSpecName "kube-api-access-v4jhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:50.619633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.619549    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.619633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.619582    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.619633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.619595    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.619633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.619608    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4jhp\" (UniqueName: \"kubernetes.io/projected/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35-kube-api-access-v4jhp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:50.832844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.832755    5551 generic.go:356] "Generic (PLEG): container finished" podID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerID="05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef" exitCode=0
Oct 31 19:00:50.832844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.832808    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"
Oct 31 19:00:50.832844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.832831    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" event={"ID":"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35","Type":"ContainerDied","Data":"05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef"}
Oct 31 19:00:50.833070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.832872    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4" event={"ID":"8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35","Type":"ContainerDied","Data":"629df6261f0c05dbea9d4c428ccb215123b13e367e0566936e98399bb324c082"}
Oct 31 19:00:50.833070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.832892    5551 scope.go:122] "RemoveContainer" containerID="05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef"
Oct 31 19:00:50.846833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.846809    5551 scope.go:122] "RemoveContainer" containerID="a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5"
Oct 31 19:00:50.858780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.858751    5551 scope.go:122] "RemoveContainer" containerID="05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef"
Oct 31 19:00:50.859161 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:00:50.859136    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef\": container with ID starting with 05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef not found: ID does not exist" containerID="05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef"
Oct 31 19:00:50.859279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.859169    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef"} err="failed to get container status \"05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef\": rpc error: code = NotFound desc = could not find container \"05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef\": container with ID starting with 05151a453cf7ad2808fe4eb951ca8d8ca6f36e8ef5b06caea0f3937940b4d5ef not found: ID does not exist"
Oct 31 19:00:50.859279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.859187    5551 scope.go:122] "RemoveContainer" containerID="a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5"
Oct 31 19:00:50.859500 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:00:50.859475    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5\": container with ID starting with a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5 not found: ID does not exist" containerID="a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5"
Oct 31 19:00:50.859584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.859506    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5"} err="failed to get container status \"a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5\": rpc error: code = NotFound desc = could not find container \"a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5\": container with ID starting with a8916f6f541a829da14a0e550b9a10ed60f4bca167e2b7ddbb0017b019e53af5 not found: ID does not exist"
Oct 31 19:00:50.997549 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:50.997513    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"]
Oct 31 19:00:51.000311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:51.000286    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-35566a-openshift-default-bb59b8757-wbxp4"]
Oct 31 19:00:52.909207 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:00:52.909174    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" path="/var/lib/kubelet/pods/8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35/volumes"
Oct 31 19:01:08.154848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.154805    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"]
Oct 31 19:01:08.155365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.155071    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc" podUID="1aa3b956-def9-40c3-895e-56988eb0abfc" containerName="maas-api" containerID="cri-o://b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5" gracePeriod=30
Oct 31 19:01:08.424827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.424763    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 19:01:08.531643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.531184    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa3b956-def9-40c3-895e-56988eb0abfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3b956-def9-40c3-895e-56988eb0abfc-openshift-service-ca\") pod \"1aa3b956-def9-40c3-895e-56988eb0abfc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") "
Oct 31 19:01:08.531643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.531369    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa3b956-def9-40c3-895e-56988eb0abfc-kube-api-access-f6vhk\" (UniqueName: \"kubernetes.io/projected/1aa3b956-def9-40c3-895e-56988eb0abfc-kube-api-access-f6vhk\") pod \"1aa3b956-def9-40c3-895e-56988eb0abfc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") "
Oct 31 19:01:08.531643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.531514    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-metrics-certs\") pod \"1aa3b956-def9-40c3-895e-56988eb0abfc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") "
Oct 31 19:01:08.531643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.531552    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls\") pod \"1aa3b956-def9-40c3-895e-56988eb0abfc\" (UID: \"1aa3b956-def9-40c3-895e-56988eb0abfc\") "
Oct 31 19:01:08.533649 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.533575    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa3b956-def9-40c3-895e-56988eb0abfc-openshift-service-ca" pod "1aa3b956-def9-40c3-895e-56988eb0abfc" (UID: "1aa3b956-def9-40c3-895e-56988eb0abfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:08.534347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.534313    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3b956-def9-40c3-895e-56988eb0abfc-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:08.561658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.560699    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls" pod "1aa3b956-def9-40c3-895e-56988eb0abfc" (UID: "1aa3b956-def9-40c3-895e-56988eb0abfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:01:08.578699 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.578657    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa3b956-def9-40c3-895e-56988eb0abfc-kube-api-access-f6vhk" pod "1aa3b956-def9-40c3-895e-56988eb0abfc" (UID: "1aa3b956-def9-40c3-895e-56988eb0abfc"). InnerVolumeSpecName "kube-api-access-f6vhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:08.578865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.578816    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-metrics-certs" pod "1aa3b956-def9-40c3-895e-56988eb0abfc" (UID: "1aa3b956-def9-40c3-895e-56988eb0abfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:01:08.635323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.635291    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:08.635323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.635320    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3b956-def9-40c3-895e-56988eb0abfc-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:08.635531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.635351    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6vhk\" (UniqueName: \"kubernetes.io/projected/1aa3b956-def9-40c3-895e-56988eb0abfc-kube-api-access-f6vhk\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:08.955008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.954969    5551 generic.go:356] "Generic (PLEG): container finished" podID="1aa3b956-def9-40c3-895e-56988eb0abfc" containerID="b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5" exitCode=0
Oct 31 19:01:08.955186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.955029    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"
Oct 31 19:01:08.955186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.955049    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc" event={"ID":"1aa3b956-def9-40c3-895e-56988eb0abfc","Type":"ContainerDied","Data":"b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5"}
Oct 31 19:01:08.955186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.955114    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc" event={"ID":"1aa3b956-def9-40c3-895e-56988eb0abfc","Type":"ContainerDied","Data":"ec7dd9d41510e2f12d669c25d05a6ae78b25c983f3371311b739955cf6e10d78"}
Oct 31 19:01:08.955186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.955146    5551 scope.go:122] "RemoveContainer" containerID="b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5"
Oct 31 19:01:08.965978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.965954    5551 scope.go:122] "RemoveContainer" containerID="b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5"
Oct 31 19:01:08.966294 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:08.966273    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5\": container with ID starting with b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5 not found: ID does not exist" containerID="b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5"
Oct 31 19:01:08.966368 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:08.966304    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5"} err="failed to get container status \"b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5\": rpc error: code = NotFound desc = could not find container \"b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5\": container with ID starting with b782a50b1eb422455fbcb1cc9889d7199aa250e900e337de3fcec527d0d84db5 not found: ID does not exist"
Oct 31 19:01:09.125622 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:09.125570    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"]
Oct 31 19:01:09.128377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:09.128343    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-919b6e-7f46b9c958-mxjbc"]
Oct 31 19:01:10.909048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:10.909015    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa3b956-def9-40c3-895e-56988eb0abfc" path="/var/lib/kubelet/pods/1aa3b956-def9-40c3-895e-56988eb0abfc/volumes"
Oct 31 19:01:12.449063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:12.449024    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"]
Oct 31 19:01:12.449544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:12.449228    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy" containerID="cri-o://148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67" gracePeriod=30
Oct 31 19:01:16.605137 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:16.605091    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:01:16.854452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:16.854408    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:01:16.854694 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:16.854527    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:01:17.732950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.732925    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 19:01:17.750489 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750441    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-certs\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750535    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-envoy\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750585    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-socket\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750696 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750673    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-kube-api-access-wrbj8\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-kube-api-access-wrbj8\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750684    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-certs" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:17.750741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750700    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-credential-socket\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750789    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-token\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-token\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750827    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-podinfo\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750864    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-data\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750932    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istiod-ca-cert\") pod \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\" (UID: \"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d\") "
Oct 31 19:01:17.750950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750933    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-socket" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:17.751111 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.750948    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-credential-socket" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:17.751183 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.751149    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-data" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:17.751443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.751394    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istiod-ca-cert" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:17.751671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.751422    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:17.751671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.751489    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:17.751671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.751555    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:17.751671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.751649    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:17.753310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.753285    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-envoy" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:17.753579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.753552    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-podinfo" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:01:17.753687 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.753627    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-kube-api-access-wrbj8" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "kube-api-access-wrbj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:17.753989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.753967    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-token" pod "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" (UID: "c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:17.852421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.852393    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrbj8\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-kube-api-access-wrbj8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:17.852421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.852420    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:17.852595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.852432    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:17.852595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.852440    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:17.852595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:17.852448    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:18.021356 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.021266    5551 generic.go:356] "Generic (PLEG): container finished" podID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerID="148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67" exitCode=0
Oct 31 19:01:18.021356 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.021313    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"
Oct 31 19:01:18.021605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.021361    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" event={"ID":"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d","Type":"ContainerDied","Data":"148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67"}
Oct 31 19:01:18.021605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.021419    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr" event={"ID":"c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d","Type":"ContainerDied","Data":"83772de904c07b8d6b834888c5f1bf895785f4b61fe98a7215d50eca503f9a29"}
Oct 31 19:01:18.021605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.021451    5551 scope.go:122] "RemoveContainer" containerID="148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67"
Oct 31 19:01:18.032867 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.032849    5551 scope.go:122] "RemoveContainer" containerID="7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7"
Oct 31 19:01:18.041909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.041890    5551 scope.go:122] "RemoveContainer" containerID="148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67"
Oct 31 19:01:18.042179 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:18.042162    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67\": container with ID starting with 148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67 not found: ID does not exist" containerID="148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67"
Oct 31 19:01:18.042227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.042189    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67"} err="failed to get container status \"148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67\": rpc error: code = NotFound desc = could not find container \"148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67\": container with ID starting with 148c22435389b05de768a5c5f72ec8ca17a1def985d822b1157f6f9b70798e67 not found: ID does not exist"
Oct 31 19:01:18.042227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.042206    5551 scope.go:122] "RemoveContainer" containerID="7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7"
Oct 31 19:01:18.042505 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:18.042482    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7\": container with ID starting with 7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7 not found: ID does not exist" containerID="7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7"
Oct 31 19:01:18.042581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.042511    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7"} err="failed to get container status \"7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7\": rpc error: code = NotFound desc = could not find container \"7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7\": container with ID starting with 7037c32e0dedb9fbe48cb82b61072f3a25c959269870793068adf6eb324b76e7 not found: ID does not exist"
Oct 31 19:01:18.187902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.187608    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"]
Oct 31 19:01:18.191140 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.191100    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-919b6e-openshift-default-66ff456dd5-s6xlr"]
Oct 31 19:01:18.908890 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:18.908856    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" path="/var/lib/kubelet/pods/c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d/volumes"
Oct 31 19:01:40.173568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.173532    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:01:40.180305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.180278    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:01:40.617703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.617621    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"]
Oct 31 19:01:40.617917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.617869    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c" podUID="6e49e7ca-b485-4395-865d-2f8886165a7e" containerName="maas-api" containerID="cri-o://1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59" gracePeriod=30
Oct 31 19:01:40.881136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.881114    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 19:01:40.966152 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.966119    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e49e7ca-b485-4395-865d-2f8886165a7e-kube-api-access-t58zq\" (UniqueName: \"kubernetes.io/projected/6e49e7ca-b485-4395-865d-2f8886165a7e-kube-api-access-t58zq\") pod \"6e49e7ca-b485-4395-865d-2f8886165a7e\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") "
Oct 31 19:01:40.966368 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.966188    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e49e7ca-b485-4395-865d-2f8886165a7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e49e7ca-b485-4395-865d-2f8886165a7e-openshift-service-ca\") pod \"6e49e7ca-b485-4395-865d-2f8886165a7e\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") "
Oct 31 19:01:40.966368 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.966314    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-metrics-certs\") pod \"6e49e7ca-b485-4395-865d-2f8886165a7e\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") "
Oct 31 19:01:40.966514 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.966381    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls\") pod \"6e49e7ca-b485-4395-865d-2f8886165a7e\" (UID: \"6e49e7ca-b485-4395-865d-2f8886165a7e\") "
Oct 31 19:01:40.966772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.966722    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e49e7ca-b485-4395-865d-2f8886165a7e-openshift-service-ca" pod "6e49e7ca-b485-4395-865d-2f8886165a7e" (UID: "6e49e7ca-b485-4395-865d-2f8886165a7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:40.967138 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.967119    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e49e7ca-b485-4395-865d-2f8886165a7e-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:40.968896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.968871    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e49e7ca-b485-4395-865d-2f8886165a7e-kube-api-access-t58zq" pod "6e49e7ca-b485-4395-865d-2f8886165a7e" (UID: "6e49e7ca-b485-4395-865d-2f8886165a7e"). InnerVolumeSpecName "kube-api-access-t58zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:40.969045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.969018    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-metrics-certs" pod "6e49e7ca-b485-4395-865d-2f8886165a7e" (UID: "6e49e7ca-b485-4395-865d-2f8886165a7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:01:40.969194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:40.969172    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls" pod "6e49e7ca-b485-4395-865d-2f8886165a7e" (UID: "6e49e7ca-b485-4395-865d-2f8886165a7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:01:41.067868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.067837    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:41.067868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.067868    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e49e7ca-b485-4395-865d-2f8886165a7e-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:41.068080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.067882    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t58zq\" (UniqueName: \"kubernetes.io/projected/6e49e7ca-b485-4395-865d-2f8886165a7e-kube-api-access-t58zq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:41.191604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.191483    5551 generic.go:356] "Generic (PLEG): container finished" podID="6e49e7ca-b485-4395-865d-2f8886165a7e" containerID="1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59" exitCode=0
Oct 31 19:01:41.191604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.191558    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c" event={"ID":"6e49e7ca-b485-4395-865d-2f8886165a7e","Type":"ContainerDied","Data":"1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59"}
Oct 31 19:01:41.191604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.191586    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"
Oct 31 19:01:41.192388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.191644    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c" event={"ID":"6e49e7ca-b485-4395-865d-2f8886165a7e","Type":"ContainerDied","Data":"800586553f873fe5f562ebf04bda98ba1c02489824c24f1a809f4ff52d14c0f4"}
Oct 31 19:01:41.192388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.191680    5551 scope.go:122] "RemoveContainer" containerID="1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59"
Oct 31 19:01:41.204638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.204611    5551 scope.go:122] "RemoveContainer" containerID="1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59"
Oct 31 19:01:41.204970 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:41.204947    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59\": container with ID starting with 1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59 not found: ID does not exist" containerID="1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59"
Oct 31 19:01:41.205022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.204980    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59"} err="failed to get container status \"1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59\": rpc error: code = NotFound desc = could not find container \"1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59\": container with ID starting with 1f5b673da13b4660dedeb2a8d0d9dc25ec8e6f366829792f2a21b7029ddbbe59 not found: ID does not exist"
Oct 31 19:01:41.354692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.354643    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"]
Oct 31 19:01:41.357690 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:41.357664    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4157da-59c449fdd5-p2p2c"]
Oct 31 19:01:42.115975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.115946    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"]
Oct 31 19:01:42.116478 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116451    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy"
Oct 31 19:01:42.116534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116479    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy"
Oct 31 19:01:42.116534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116494    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy"
Oct 31 19:01:42.116534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116500    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy"
Oct 31 19:01:42.116534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116510    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy"
Oct 31 19:01:42.116534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116516    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy"
Oct 31 19:01:42.116534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116526    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy"
Oct 31 19:01:42.116534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116530    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116547    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6e49e7ca-b485-4395-865d-2f8886165a7e" containerName="maas-api"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116551    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e49e7ca-b485-4395-865d-2f8886165a7e" containerName="maas-api"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116562    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116568    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116575    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1aa3b956-def9-40c3-895e-56988eb0abfc" containerName="maas-api"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116579    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa3b956-def9-40c3-895e-56988eb0abfc" containerName="maas-api"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116663    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116671    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116677    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116684    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed65c6e-fc1f-4857-9ad0-2ebfd1d56e35" containerName="istio-proxy"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116692    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e49e7ca-b485-4395-865d-2f8886165a7e" containerName="maas-api"
Oct 31 19:01:42.116747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.116699    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa3b956-def9-40c3-895e-56988eb0abfc" containerName="maas-api"
Oct 31 19:01:42.121134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.121118    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.173316 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.173238    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4c58c5-serving-cert\""
Oct 31 19:01:42.177502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.177476    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"]
Oct 31 19:01:42.181902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.181871    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.182065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.181952    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-metrics-certs\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.182065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.182008    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.182065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.182061    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzs2g\" (UniqueName: \"kubernetes.io/projected/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-kube-api-access-pzs2g\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.283546 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.283515    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.284014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.283565    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-metrics-certs\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.284014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.283600    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.284014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.283630    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzs2g\" (UniqueName: \"kubernetes.io/projected/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-kube-api-access-pzs2g\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.284014 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:42.283686    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-serving-cert: secret "maas-api-e2e-ipp-b-4c58c5-serving-cert" not found
Oct 31 19:01:42.284014 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:42.283767    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls podName:7477fcef-3dfe-415c-b8f7-80c2cc3cd602 nodeName:}" failed. No retries permitted until 2026-10-31 19:01:42.78374353 +0000 UTC m=+2406.014981818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls") pod "maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q" (UID: "7477fcef-3dfe-415c-b8f7-80c2cc3cd602") : secret "maas-api-e2e-ipp-b-4c58c5-serving-cert" not found
Oct 31 19:01:42.284639 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.284613    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.286406 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.286379    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-metrics-certs\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.340327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.340297    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzs2g\" (UniqueName: \"kubernetes.io/projected/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-kube-api-access-pzs2g\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.789065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.789036    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.791756 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.791731    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:42.909420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:42.909384    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e49e7ca-b485-4395-865d-2f8886165a7e" path="/var/lib/kubelet/pods/6e49e7ca-b485-4395-865d-2f8886165a7e/volumes"
Oct 31 19:01:43.034280 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:43.034248    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:43.415752 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:43.415724    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"]
Oct 31 19:01:43.417060 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:01:43.417030    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7477fcef_3dfe_415c_b8f7_80c2cc3cd602.slice/crio-29ce0c4114ca0054bc24a78d9e74939cfcc2831bfbf9bd59b5294573e21f81a2 WatchSource:0}: Error finding container 29ce0c4114ca0054bc24a78d9e74939cfcc2831bfbf9bd59b5294573e21f81a2: Status 404 returned error can't find the container with id 29ce0c4114ca0054bc24a78d9e74939cfcc2831bfbf9bd59b5294573e21f81a2
Oct 31 19:01:44.217734 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:44.217694    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q" event={"ID":"7477fcef-3dfe-415c-b8f7-80c2cc3cd602","Type":"ContainerStarted","Data":"29ce0c4114ca0054bc24a78d9e74939cfcc2831bfbf9bd59b5294573e21f81a2"}
Oct 31 19:01:45.226141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:45.226099    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q" event={"ID":"7477fcef-3dfe-415c-b8f7-80c2cc3cd602","Type":"ContainerStarted","Data":"bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d"}
Oct 31 19:01:45.226670 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:45.226397    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:45.340139 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:45.340080    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q" podStartSLOduration=2.319644835 podStartE2EDuration="3.340065955s" totalImagesPullingTime="1.02042112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:01:45.338380602 +0000 UTC m=+2408.569618915" watchObservedRunningTime="2026-10-31 19:01:45.340065955 +0000 UTC m=+2408.571304244"
Oct 31 19:01:46.604043 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:46.603999    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:01:48.952115 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:48.952081    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"]
Oct 31 19:01:48.952616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:48.952282    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy" containerID="cri-o://0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f" gracePeriod=30
Oct 31 19:01:51.238880 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:51.238850    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:01:54.214637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.214613    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 19:01:54.245142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245079    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-kube-api-access-xvjbc\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-kube-api-access-xvjbc\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245116    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a940ce0-f138-492c-84b6-aaf10d45a459-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a940ce0-f138-492c-84b6-aaf10d45a459-istiod-ca-cert\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245334 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245148    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-certs\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245334 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245282    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-credential-socket\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245491 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245358    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-podinfo\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245491 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245435    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-certs" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:54.245620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245506    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-socket\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245553    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-envoy\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245591    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-data\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245596    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-credential-socket" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:54.245820 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245633    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a940ce0-f138-492c-84b6-aaf10d45a459-istiod-ca-cert" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:54.245820 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245643    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-token\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-token\") pod \"0a940ce0-f138-492c-84b6-aaf10d45a459\" (UID: \"0a940ce0-f138-492c-84b6-aaf10d45a459\") "
Oct 31 19:01:54.245820 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245742    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-socket" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:54.245959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.245888    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-data" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:54.246323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.246231    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a940ce0-f138-492c-84b6-aaf10d45a459-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.246323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.246257    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.246323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.246273    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.246323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.246286    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.246323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.246301    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.247689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.247659    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-kube-api-access-xvjbc" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "kube-api-access-xvjbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:54.247689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.247664    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-podinfo" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:01:54.247813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.247738    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-envoy" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:01:54.247970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.247953    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-token" pod "0a940ce0-f138-492c-84b6-aaf10d45a459" (UID: "0a940ce0-f138-492c-84b6-aaf10d45a459"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:54.293646 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.293613    5551 generic.go:356] "Generic (PLEG): container finished" podID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerID="0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f" exitCode=0
Oct 31 19:01:54.293790 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.293684    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" event={"ID":"0a940ce0-f138-492c-84b6-aaf10d45a459","Type":"ContainerDied","Data":"0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f"}
Oct 31 19:01:54.293790 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.293775    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct" event={"ID":"0a940ce0-f138-492c-84b6-aaf10d45a459","Type":"ContainerDied","Data":"bea2321e8cfe762ec6ae6c5014ed4bdf0249b2888de42a728e34ee4524cab456"}
Oct 31 19:01:54.293898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.293813    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"
Oct 31 19:01:54.293950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.293804    5551 scope.go:122] "RemoveContainer" containerID="0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f"
Oct 31 19:01:54.304391 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.304374    5551 scope.go:122] "RemoveContainer" containerID="aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f"
Oct 31 19:01:54.313016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.312993    5551 scope.go:122] "RemoveContainer" containerID="0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f"
Oct 31 19:01:54.313272 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:54.313254    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f\": container with ID starting with 0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f not found: ID does not exist" containerID="0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f"
Oct 31 19:01:54.313323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.313281    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f"} err="failed to get container status \"0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f\": rpc error: code = NotFound desc = could not find container \"0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f\": container with ID starting with 0464897dbc0613b958626de1fab5395e08f1763fefa3327cf4356cf7ccb7fc6f not found: ID does not exist"
Oct 31 19:01:54.313323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.313299    5551 scope.go:122] "RemoveContainer" containerID="aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f"
Oct 31 19:01:54.313558 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:54.313542    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f\": container with ID starting with aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f not found: ID does not exist" containerID="aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f"
Oct 31 19:01:54.313618 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.313561    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f"} err="failed to get container status \"aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f\": rpc error: code = NotFound desc = could not find container \"aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f\": container with ID starting with aead2fc8c4692c329f9e691af37c3c319122fb27de180893c7a2853298c9cc6f not found: ID does not exist"
Oct 31 19:01:54.347205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.347180    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.347205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.347202    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.347338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.347232    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvjbc\" (UniqueName: \"kubernetes.io/projected/0a940ce0-f138-492c-84b6-aaf10d45a459-kube-api-access-xvjbc\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.347338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.347242    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a940ce0-f138-492c-84b6-aaf10d45a459-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:54.492574 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.492520    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"]
Oct 31 19:01:54.496429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.496364    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4157da-openshift-default-679dccb777-pztct"]
Oct 31 19:01:54.909031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:54.908999    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" path="/var/lib/kubelet/pods/0a940ce0-f138-492c-84b6-aaf10d45a459/volumes"
Oct 31 19:01:56.571967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.571930    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"]
Oct 31 19:01:56.572401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.572181    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb" podUID="b7dc1489-f35d-4533-a74a-796288531f74" containerName="maas-api" containerID="cri-o://337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62" gracePeriod=30
Oct 31 19:01:56.870276 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.870246    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 19:01:56.977757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.977715    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-metrics-certs\") pod \"b7dc1489-f35d-4533-a74a-796288531f74\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") "
Oct 31 19:01:56.977929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.977801    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7dc1489-f35d-4533-a74a-796288531f74-kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/b7dc1489-f35d-4533-a74a-796288531f74-kube-api-access-pbm6m\") pod \"b7dc1489-f35d-4533-a74a-796288531f74\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") "
Oct 31 19:01:56.977929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.977829    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls\") pod \"b7dc1489-f35d-4533-a74a-796288531f74\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") "
Oct 31 19:01:56.978061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.978035    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7dc1489-f35d-4533-a74a-796288531f74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc1489-f35d-4533-a74a-796288531f74-openshift-service-ca\") pod \"b7dc1489-f35d-4533-a74a-796288531f74\" (UID: \"b7dc1489-f35d-4533-a74a-796288531f74\") "
Oct 31 19:01:56.978698 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.978607    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7dc1489-f35d-4533-a74a-796288531f74-openshift-service-ca" pod "b7dc1489-f35d-4533-a74a-796288531f74" (UID: "b7dc1489-f35d-4533-a74a-796288531f74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:56.980963 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.980914    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls" pod "b7dc1489-f35d-4533-a74a-796288531f74" (UID: "b7dc1489-f35d-4533-a74a-796288531f74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:01:56.980963 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.980935    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-metrics-certs" pod "b7dc1489-f35d-4533-a74a-796288531f74" (UID: "b7dc1489-f35d-4533-a74a-796288531f74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:01:56.981154 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:56.981118    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7dc1489-f35d-4533-a74a-796288531f74-kube-api-access-pbm6m" pod "b7dc1489-f35d-4533-a74a-796288531f74" (UID: "b7dc1489-f35d-4533-a74a-796288531f74"). InnerVolumeSpecName "kube-api-access-pbm6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:57.079492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.079439    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc1489-f35d-4533-a74a-796288531f74-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:57.079492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.079485    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:57.079492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.079494    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/b7dc1489-f35d-4533-a74a-796288531f74-kube-api-access-pbm6m\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:57.079492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.079505    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7dc1489-f35d-4533-a74a-796288531f74-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:57.314464 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.314430    5551 generic.go:356] "Generic (PLEG): container finished" podID="b7dc1489-f35d-4533-a74a-796288531f74" containerID="337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62" exitCode=0
Oct 31 19:01:57.314642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.314512    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"
Oct 31 19:01:57.314642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.314524    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb" event={"ID":"b7dc1489-f35d-4533-a74a-796288531f74","Type":"ContainerDied","Data":"337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62"}
Oct 31 19:01:57.314642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.314579    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb" event={"ID":"b7dc1489-f35d-4533-a74a-796288531f74","Type":"ContainerDied","Data":"89b4402c31645782cb64c71d620e6ce6a7a0792f9c046c7657ddb54cd8034831"}
Oct 31 19:01:57.314642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.314600    5551 scope.go:122] "RemoveContainer" containerID="337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62"
Oct 31 19:01:57.326044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.326027    5551 scope.go:122] "RemoveContainer" containerID="337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62"
Oct 31 19:01:57.326329 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:01:57.326310    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62\": container with ID starting with 337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62 not found: ID does not exist" containerID="337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62"
Oct 31 19:01:57.326383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.326341    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62"} err="failed to get container status \"337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62\": rpc error: code = NotFound desc = could not find container \"337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62\": container with ID starting with 337e4235fcf9705f92952adad5ac504a0f48b0144e5ce21b7ec9093b5bd80e62 not found: ID does not exist"
Oct 31 19:01:57.478829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.478791    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"]
Oct 31 19:01:57.481325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:57.481298    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3932ef-64898964ff-h9krb"]
Oct 31 19:01:58.915093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:01:58.915057    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7dc1489-f35d-4533-a74a-796288531f74" path="/var/lib/kubelet/pods/b7dc1489-f35d-4533-a74a-796288531f74/volumes"
Oct 31 19:02:00.431132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:00.431100    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"]
Oct 31 19:02:00.431681 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:00.431369    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy" containerID="cri-o://c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e" gracePeriod=30
Oct 31 19:02:05.095029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.094961    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:02:05.095557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.095027    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:02:05.787650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.787624    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 19:02:05.905411 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905376    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-data\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.905638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905435    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba30e08-8922-4c58-bf01-812abde01f55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba30e08-8922-4c58-bf01-812abde01f55-istiod-ca-cert\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.905638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905498    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ba30e08-8922-4c58-bf01-812abde01f55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba30e08-8922-4c58-bf01-812abde01f55-istio-podinfo\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.905638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905535    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-istio-token\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-istio-token\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.905638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905593    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-envoy\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.905875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905642    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-certs\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.905875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905680    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-socket\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.905875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905700    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-data" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:05.905875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905713    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-credential-socket\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.905875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905809    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-kube-api-access-qc4hn\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-kube-api-access-qc4hn\") pod \"5ba30e08-8922-4c58-bf01-812abde01f55\" (UID: \"5ba30e08-8922-4c58-bf01-812abde01f55\") "
Oct 31 19:02:05.906156 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905894    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba30e08-8922-4c58-bf01-812abde01f55-istiod-ca-cert" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:02:05.906156 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.905932    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-credential-socket" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:05.906269 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.906180    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-certs" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:05.906269 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.906246    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:05.906269 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.906267    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:05.906526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.906281    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba30e08-8922-4c58-bf01-812abde01f55-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:05.906526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.906387    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-socket" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:05.909596 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.909555    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-envoy" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:05.912258 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.912234    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-kube-api-access-qc4hn" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "kube-api-access-qc4hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:05.912428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.912246    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-istio-token" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:05.912610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:05.912584    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba30e08-8922-4c58-bf01-812abde01f55-istio-podinfo" pod "5ba30e08-8922-4c58-bf01-812abde01f55" (UID: "5ba30e08-8922-4c58-bf01-812abde01f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:02:06.010534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.008427    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.010534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.008493    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.010534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.008510    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba30e08-8922-4c58-bf01-812abde01f55-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.010534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.008523    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc4hn\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-kube-api-access-qc4hn\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.010534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.008541    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba30e08-8922-4c58-bf01-812abde01f55-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.010534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.008556    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba30e08-8922-4c58-bf01-812abde01f55-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.397793 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.397755    5551 generic.go:356] "Generic (PLEG): container finished" podID="5ba30e08-8922-4c58-bf01-812abde01f55" containerID="c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e" exitCode=0
Oct 31 19:02:06.398307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.398284    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"
Oct 31 19:02:06.398440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.398417    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" event={"ID":"5ba30e08-8922-4c58-bf01-812abde01f55","Type":"ContainerDied","Data":"c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e"}
Oct 31 19:02:06.398599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.398583    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6" event={"ID":"5ba30e08-8922-4c58-bf01-812abde01f55","Type":"ContainerDied","Data":"cabef3107507446af82f3b6e8c3c68f410e72366d8d62c4ff010af9e8978ab8d"}
Oct 31 19:02:06.398701 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.398689    5551 scope.go:122] "RemoveContainer" containerID="c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e"
Oct 31 19:02:06.415287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.415243    5551 scope.go:122] "RemoveContainer" containerID="aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e"
Oct 31 19:02:06.424893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.424840    5551 scope.go:122] "RemoveContainer" containerID="c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e"
Oct 31 19:02:06.425188 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:02:06.425167    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e\": container with ID starting with c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e not found: ID does not exist" containerID="c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e"
Oct 31 19:02:06.425332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.425309    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e"} err="failed to get container status \"c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e\": rpc error: code = NotFound desc = could not find container \"c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e\": container with ID starting with c569f1657896a2533413fc621dc8eb0fa699cd78b53eabac1af073dc37bdc07e not found: ID does not exist"
Oct 31 19:02:06.425434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.425423    5551 scope.go:122] "RemoveContainer" containerID="aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e"
Oct 31 19:02:06.425816 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:02:06.425781    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e\": container with ID starting with aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e not found: ID does not exist" containerID="aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e"
Oct 31 19:02:06.425898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.425822    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e"} err="failed to get container status \"aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e\": rpc error: code = NotFound desc = could not find container \"aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e\": container with ID starting with aafe7dfb8650ddf420c1b3254435bdea87888bd8958eb0420c553f0deab2811e not found: ID does not exist"
Oct 31 19:02:06.584971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.584938    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"]
Oct 31 19:02:06.587383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.587357    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3932ef-openshift-default-695f857c89-6dlh6"]
Oct 31 19:02:06.910784 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:06.910749    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" path="/var/lib/kubelet/pods/5ba30e08-8922-4c58-bf01-812abde01f55/volumes"
Oct 31 19:02:10.371320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.371266    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"]
Oct 31 19:02:10.372141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372117    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b7dc1489-f35d-4533-a74a-796288531f74" containerName="maas-api"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372145    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7dc1489-f35d-4533-a74a-796288531f74" containerName="maas-api"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372160    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372168    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372192    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372200    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372214    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372223    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372241    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372248    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372265    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy"
Oct 31 19:02:10.372281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372272    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy"
Oct 31 19:02:10.372902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372409    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce15580-1369-48c7-b20e-cec1a82cb6ed" containerName="istio-proxy"
Oct 31 19:02:10.372902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372421    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bbd9f0-9646-4c5b-bf7f-a9b8a7b6e41d" containerName="istio-proxy"
Oct 31 19:02:10.372902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372434    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy"
Oct 31 19:02:10.372902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372472    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7dc1489-f35d-4533-a74a-796288531f74" containerName="maas-api"
Oct 31 19:02:10.372902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372481    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy"
Oct 31 19:02:10.372902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.372493    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a940ce0-f138-492c-84b6-aaf10d45a459" containerName="istio-proxy"
Oct 31 19:02:10.377490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.377449    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464084 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464008    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464084 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464057    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32598a29-4328-48fd-98dc-f896ef2d96ed-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464124    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464158    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464229    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464265    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464323    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7nz\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-kube-api-access-qw7nz\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464354    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.464664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.464380    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565316    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565423    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565503    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565565    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7nz\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-kube-api-access-qw7nz\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565599    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565629    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565741    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565767    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32598a29-4328-48fd-98dc-f896ef2d96ed-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.565842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565832    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.566062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.565906    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.566512 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.566419    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.566657 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.566569    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.567387 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.566923    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.567387 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.567348    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32598a29-4328-48fd-98dc-f896ef2d96ed-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.569267 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.569233    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.569602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.569554    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.576181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.576151    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"]
Oct 31 19:02:10.622112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.622045    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.622249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.622220    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7nz\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-kube-api-access-qw7nz\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:10.697946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:10.697908    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:11.138557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:11.138528    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"]
Oct 31 19:02:11.140617 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:02:11.140591    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32598a29_4328_48fd_98dc_f896ef2d96ed.slice/crio-6de15151fdaa67a08820af61b27721e882ebea4896d968808181ac6892d94146 WatchSource:0}: Error finding container 6de15151fdaa67a08820af61b27721e882ebea4896d968808181ac6892d94146: Status 404 returned error can't find the container with id 6de15151fdaa67a08820af61b27721e882ebea4896d968808181ac6892d94146
Oct 31 19:02:11.143217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:11.143181    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:11.143302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:11.143264    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:11.143349 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:11.143297    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:11.435613 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:11.435512    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" event={"ID":"32598a29-4328-48fd-98dc-f896ef2d96ed","Type":"ContainerStarted","Data":"848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b"}
Oct 31 19:02:11.435613 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:11.435580    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" event={"ID":"32598a29-4328-48fd-98dc-f896ef2d96ed","Type":"ContainerStarted","Data":"6de15151fdaa67a08820af61b27721e882ebea4896d968808181ac6892d94146"}
Oct 31 19:02:11.553132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:11.553056    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" podStartSLOduration=1.553033729 podStartE2EDuration="1.553033729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:11.551207834 +0000 UTC m=+2434.782446126" watchObservedRunningTime="2026-10-31 19:02:11.553033729 +0000 UTC m=+2434.784272021"
Oct 31 19:02:11.698918 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:11.698847    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:12.017104 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.017029    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"]
Oct 31 19:02:12.017679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.017663    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba30e08-8922-4c58-bf01-812abde01f55" containerName="istio-proxy"
Oct 31 19:02:12.021098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.021079    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.082642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.082608    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"]
Oct 31 19:02:12.101136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101098    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2x2d\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-kube-api-access-k2x2d\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.101508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101428    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.101599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101510    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.101599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101560    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.101732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101595    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.101732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101669    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.101837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101749    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.101837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101783    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.101837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.101810    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.124140 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.124112    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a8587d-openshift-default-dockercfg-zdp6b\""
Oct 31 19:02:12.202721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.202684    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.202721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.202721    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.202984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.202750    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.202984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.202786    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2x2d\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-kube-api-access-k2x2d\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.202984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.202909    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.202984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.202940    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.203194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.202988    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.203194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.203021    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.203194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.203090    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.203440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.203413    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.203550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.203531    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.203607 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.203565    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.203907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.203885    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.204038 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.203916    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.205921 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.205886    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.206019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.205992    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.274347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.274236    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.291861 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.291789    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2x2d\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-kube-api-access-k2x2d\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.336507 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.336451    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:12.700536 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.700496    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 19:02:12.701948 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.700561    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 19:02:12.760479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.760356    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": read tcp 10.134.0.2:33578->10.134.0.138:15021: read: connection reset by peer" start-of-body=
Oct 31 19:02:12.760479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.760413    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": read tcp 10.134.0.2:33578->10.134.0.138:15021: read: connection reset by peer"
Oct 31 19:02:12.801875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.801843    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"]
Oct 31 19:02:12.882987 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.882010    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:12.882987 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.882101    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:12.882987 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:12.882142    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:13.473483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.472084    5551 generic.go:356] "Generic (PLEG): container finished" podID="762f8ee8-681f-495c-b902-75a8d1214edf" containerID="38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470" exitCode=0
Oct 31 19:02:13.473483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.472315    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" event={"ID":"762f8ee8-681f-495c-b902-75a8d1214edf","Type":"ContainerDied","Data":"38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470"}
Oct 31 19:02:13.473483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.472827    5551 scope.go:122] "RemoveContainer" containerID="38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470"
Oct 31 19:02:13.491301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.491267    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" event={"ID":"f8a134d9-fd92-4d5b-9e5b-1476abfd3092","Type":"ContainerStarted","Data":"ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d"}
Oct 31 19:02:13.491585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.491522    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" event={"ID":"f8a134d9-fd92-4d5b-9e5b-1476abfd3092","Type":"ContainerStarted","Data":"b0d44398a5423de9e6a2bf664dc0e6ed7ab73c43fe098e47599457ff727bb2e0"}
Oct 31 19:02:13.492339 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.492298    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:13.492632 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.492391    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:13.492632 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.492436    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:13.703138 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.701153    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 19:02:13.703884 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.703848    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 19:02:13.715032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.707285    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podStartSLOduration=2.707268112 podStartE2EDuration="2.707268112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:13.706143172 +0000 UTC m=+2436.937381463" watchObservedRunningTime="2026-10-31 19:02:13.707268112 +0000 UTC m=+2436.938506403"
Oct 31 19:02:13.725014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.724942    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"]
Oct 31 19:02:13.729044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.729022    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.786598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.786561    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-79e7b6-serving-cert\""
Oct 31 19:02:13.834485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.824303    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-metrics-certs\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.834485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.824375    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6360aaf-34aa-46f5-af0a-5191af0e5479-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.834485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.824480    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.834485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.824537    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn945\" (UniqueName: \"kubernetes.io/projected/b6360aaf-34aa-46f5-af0a-5191af0e5479-kube-api-access-bn945\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.837968 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.835101    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"]
Oct 31 19:02:13.936481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.935708    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.936481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.935791    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn945\" (UniqueName: \"kubernetes.io/projected/b6360aaf-34aa-46f5-af0a-5191af0e5479-kube-api-access-bn945\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.936481 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:02:13.935867    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-serving-cert: secret "maas-api-e2e-praxis-a-79e7b6-serving-cert" not found
Oct 31 19:02:13.936481 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:02:13.935935    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls podName:b6360aaf-34aa-46f5-af0a-5191af0e5479 nodeName:}" failed. No retries permitted until 2026-10-31 19:02:14.435914708 +0000 UTC m=+2437.667152998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls") pod "maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8" (UID: "b6360aaf-34aa-46f5-af0a-5191af0e5479") : secret "maas-api-e2e-praxis-a-79e7b6-serving-cert" not found
Oct 31 19:02:13.936481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.935985    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-metrics-certs\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.936481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.936021    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6360aaf-34aa-46f5-af0a-5191af0e5479-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.936927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.936894    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6360aaf-34aa-46f5-af0a-5191af0e5479-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:13.941503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:13.939907    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-metrics-certs\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:14.040886 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.040792    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn945\" (UniqueName: \"kubernetes.io/projected/b6360aaf-34aa-46f5-af0a-5191af0e5479-kube-api-access-bn945\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:14.336999 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.336914    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:14.338278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.338254    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:02:14.338361 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.338303    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 19:02:14.444482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.444424    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:14.448914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.448876    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls\") pod \"maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:14.539878 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.539840    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" event={"ID":"762f8ee8-681f-495c-b902-75a8d1214edf","Type":"ContainerStarted","Data":"15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449"}
Oct 31 19:02:14.695760 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.695708    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:14.706061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.705612    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 19:02:14.706061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:14.705671    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 19:02:15.248500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:15.248129    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:02:15.248500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:15.248253    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 19:02:15.248500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:15.248294    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 19:02:15.339556 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:15.339393    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:02:15.339840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:15.339777    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 19:02:15.595541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:15.587645    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"]
Oct 31 19:02:15.701485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:15.700860    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 19:02:15.701485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:15.700926    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 19:02:16.197484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:16.196824    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 19:02:16.197484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:16.196900    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 19:02:16.337266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:16.337229    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:02:16.337589 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:16.337542    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 19:02:16.656482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:16.648927    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8" event={"ID":"b6360aaf-34aa-46f5-af0a-5191af0e5479","Type":"ContainerStarted","Data":"4d93150fdf57f80f00cced0a9aae8b2273926674741b9b30f6a18c45661b9c29"}
Oct 31 19:02:16.707319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:16.707282    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 19:02:16.707592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:16.707564    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 19:02:16.785482 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:02:16.781903    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:02:17.200542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.199802    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 19:02:17.200542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.199861    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 19:02:17.351495 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.347021    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:02:17.351495 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.347080    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 19:02:17.688508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.688469    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8" event={"ID":"b6360aaf-34aa-46f5-af0a-5191af0e5479","Type":"ContainerStarted","Data":"ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e"}
Oct 31 19:02:17.691563 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.691536    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:17.700974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.700943    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 19:02:17.701202 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.701173    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 19:02:17.811481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:17.811186    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8" podStartSLOduration=3.828546785 podStartE2EDuration="4.811152977s" totalImagesPullingTime="982.606192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:17.809998971 +0000 UTC m=+2441.041237261" watchObservedRunningTime="2026-10-31 19:02:17.811152977 +0000 UTC m=+2441.042391268"
Oct 31 19:02:18.199907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:18.199866    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 19:02:18.200232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:18.200197    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 19:02:18.337305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:18.337277    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:02:18.337784 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:18.337760    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 19:02:18.706766 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:18.706736    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:18.707178 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:18.707158    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:18.707888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:18.707868    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:02:19.198832 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:19.198796    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 19:02:19.203205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:19.203169    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 19:02:19.337431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:19.337403    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:02:19.338016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:19.337981    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 19:02:20.286008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:20.285975    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:02:20.286602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:20.286544    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:02:20.287871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:20.287852    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:02:20.336916 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:20.336889    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:02:20.337036 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:20.336940    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 19:02:22.071679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:22.071650    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:22.076338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:22.076312    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:22.078311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:22.078288    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:02:23.711359 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:23.711322    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:02:40.367314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.367278    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"]
Oct 31 19:02:40.372611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.372576    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.485506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.485434    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.485699 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.485636    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.485771 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.485707    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.485834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.485763    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.485902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.485842    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.485902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.485881    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.486009 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.485926    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.486009 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.485963    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnk2\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-kube-api-access-whnk2\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.486123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.486027    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530506f9-6a80-4249-8c5d-890be1bb26c5-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.572426 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.572394    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"]
Oct 31 19:02:40.587103 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587070    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587184    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587249    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587271    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587320    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587359    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587405    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587442    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnk2\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-kube-api-access-whnk2\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587529    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530506f9-6a80-4249-8c5d-890be1bb26c5-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587584    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587627    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.587962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.587878    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.588131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.588108    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.588317 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.588292    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530506f9-6a80-4249-8c5d-890be1bb26c5-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.590796 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.590764    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.591093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.591070    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.644121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.644089    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.644281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.644119    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnk2\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-kube-api-access-whnk2\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:40.688846 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:40.688815    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:41.121861 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:41.121830    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"]
Oct 31 19:02:41.122790 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:02:41.122762    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530506f9_6a80_4249_8c5d_890be1bb26c5.slice/crio-f2984536d63cf18546cb98fea03997c60d586f12487aec276fe3b4a6616c3e6c WatchSource:0}: Error finding container f2984536d63cf18546cb98fea03997c60d586f12487aec276fe3b4a6616c3e6c: Status 404 returned error can't find the container with id f2984536d63cf18546cb98fea03997c60d586f12487aec276fe3b4a6616c3e6c
Oct 31 19:02:41.125282 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:41.125249    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:41.125365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:41.125315    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:41.125365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:41.125353    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:41.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:41.866168    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" event={"ID":"530506f9-6a80-4249-8c5d-890be1bb26c5","Type":"ContainerStarted","Data":"5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d"}
Oct 31 19:02:41.867310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:41.866238    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" event={"ID":"530506f9-6a80-4249-8c5d-890be1bb26c5","Type":"ContainerStarted","Data":"f2984536d63cf18546cb98fea03997c60d586f12487aec276fe3b4a6616c3e6c"}
Oct 31 19:02:42.689855 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:42.689816    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:42.691495 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:42.691432    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:02:42.691643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:42.691538    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 19:02:43.689892 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:43.689844    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:02:43.690399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:43.689917    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 19:02:44.689836 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:44.689797    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:02:44.690033 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:44.689849    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 19:02:45.689166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:45.689134    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:02:45.689364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:45.689191    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 19:02:46.606385 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:02:46.606336    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:02:46.690068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:46.690020    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:02:46.690258 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:46.690086    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 19:02:47.694252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:47.694223    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:47.694780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:47.694762    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:47.695502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:47.695481    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:02:47.860834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:47.860776    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" podStartSLOduration=7.86076249 podStartE2EDuration="7.86076249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:41.98374271 +0000 UTC m=+2465.214981002" watchObservedRunningTime="2026-10-31 19:02:47.86076249 +0000 UTC m=+2471.092000779"
Oct 31 19:02:49.448542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.448506    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"]
Oct 31 19:02:49.448971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.448725    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x" podUID="fe4cf1cb-a637-47b0-81c8-0478c86e5969" containerName="maas-api" containerID="cri-o://c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f" gracePeriod=30
Oct 31 19:02:49.719440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.719372    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 19:02:49.727703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.727668    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-metrics-certs\") pod \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") "
Oct 31 19:02:49.727843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.727713    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4cf1cb-a637-47b0-81c8-0478c86e5969-kube-api-access-zqhq8\" (UniqueName: \"kubernetes.io/projected/fe4cf1cb-a637-47b0-81c8-0478c86e5969-kube-api-access-zqhq8\") pod \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") "
Oct 31 19:02:49.727843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.727733    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4cf1cb-a637-47b0-81c8-0478c86e5969-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe4cf1cb-a637-47b0-81c8-0478c86e5969-openshift-service-ca\") pod \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") "
Oct 31 19:02:49.727843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.727823    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls\") pod \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\" (UID: \"fe4cf1cb-a637-47b0-81c8-0478c86e5969\") "
Oct 31 19:02:49.728149 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.728124    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4cf1cb-a637-47b0-81c8-0478c86e5969-openshift-service-ca" pod "fe4cf1cb-a637-47b0-81c8-0478c86e5969" (UID: "fe4cf1cb-a637-47b0-81c8-0478c86e5969"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:02:49.728401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.728325    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe4cf1cb-a637-47b0-81c8-0478c86e5969-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:49.730047 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.730019    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-metrics-certs" pod "fe4cf1cb-a637-47b0-81c8-0478c86e5969" (UID: "fe4cf1cb-a637-47b0-81c8-0478c86e5969"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:02:49.730133 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.730061    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4cf1cb-a637-47b0-81c8-0478c86e5969-kube-api-access-zqhq8" pod "fe4cf1cb-a637-47b0-81c8-0478c86e5969" (UID: "fe4cf1cb-a637-47b0-81c8-0478c86e5969"). InnerVolumeSpecName "kube-api-access-zqhq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:49.730133 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.730107    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls" pod "fe4cf1cb-a637-47b0-81c8-0478c86e5969" (UID: "fe4cf1cb-a637-47b0-81c8-0478c86e5969"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:02:49.830083 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.830049    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:49.830083 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.830077    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4cf1cb-a637-47b0-81c8-0478c86e5969-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:49.830083 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.830087    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqhq8\" (UniqueName: \"kubernetes.io/projected/fe4cf1cb-a637-47b0-81c8-0478c86e5969-kube-api-access-zqhq8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:49.931585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.931554    5551 generic.go:356] "Generic (PLEG): container finished" podID="fe4cf1cb-a637-47b0-81c8-0478c86e5969" containerID="c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f" exitCode=0
Oct 31 19:02:49.931773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.931592    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"
Oct 31 19:02:49.931773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.931638    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x" event={"ID":"fe4cf1cb-a637-47b0-81c8-0478c86e5969","Type":"ContainerDied","Data":"c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f"}
Oct 31 19:02:49.931773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.931700    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x" event={"ID":"fe4cf1cb-a637-47b0-81c8-0478c86e5969","Type":"ContainerDied","Data":"ad713b26e69f59f42c56f6b772aa169e1422e1f268347ed9ed9639dbefc3b39b"}
Oct 31 19:02:49.931773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.931728    5551 scope.go:122] "RemoveContainer" containerID="c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f"
Oct 31 19:02:49.942249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.942232    5551 scope.go:122] "RemoveContainer" containerID="c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f"
Oct 31 19:02:49.942520 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:02:49.942500    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f\": container with ID starting with c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f not found: ID does not exist" containerID="c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f"
Oct 31 19:02:49.942584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:49.942528    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f"} err="failed to get container status \"c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f\": rpc error: code = NotFound desc = could not find container \"c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f\": container with ID starting with c987b9be2d6336b30b2f3d59b9b62e89c3e5403390880c4e38c7d18ea906ed4f not found: ID does not exist"
Oct 31 19:02:50.098959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:50.098882    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"]
Oct 31 19:02:50.111686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:50.111657    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e43ccf-7b8f4f7b8-xj94x"]
Oct 31 19:02:50.910709 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:50.910677    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4cf1cb-a637-47b0-81c8-0478c86e5969" path="/var/lib/kubelet/pods/fe4cf1cb-a637-47b0-81c8-0478c86e5969/volumes"
Oct 31 19:02:55.651108 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.651075    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"]
Oct 31 19:02:55.651602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.651317    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n" podUID="adff5c77-695a-4300-af20-ee81d7fa13d7" containerName="maas-api" containerID="cri-o://8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02" gracePeriod=30
Oct 31 19:02:55.927492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.927407    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:02:55.981482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.981309    5551 generic.go:356] "Generic (PLEG): container finished" podID="adff5c77-695a-4300-af20-ee81d7fa13d7" containerID="8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02" exitCode=0
Oct 31 19:02:55.981482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.981378    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"
Oct 31 19:02:55.981482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.981379    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n" event={"ID":"adff5c77-695a-4300-af20-ee81d7fa13d7","Type":"ContainerDied","Data":"8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02"}
Oct 31 19:02:55.981810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.981524    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n" event={"ID":"adff5c77-695a-4300-af20-ee81d7fa13d7","Type":"ContainerDied","Data":"bd88273bcf2fba0814061001d49c427e4530afbd4ddbbeb83aef11dbc330c2ef"}
Oct 31 19:02:55.981810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.981577    5551 scope.go:122] "RemoveContainer" containerID="8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02"
Oct 31 19:02:55.994266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.994246    5551 scope.go:122] "RemoveContainer" containerID="8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02"
Oct 31 19:02:55.994605 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:02:55.994581    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02\": container with ID starting with 8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02 not found: ID does not exist" containerID="8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02"
Oct 31 19:02:55.994709 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:55.994620    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02"} err="failed to get container status \"8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02\": rpc error: code = NotFound desc = could not find container \"8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02\": container with ID starting with 8e6f4b860f1b4d3f658f073c89fc576f75689c9d320900a3e6ba22026804ea02 not found: ID does not exist"
Oct 31 19:02:56.011429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.011403    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adff5c77-695a-4300-af20-ee81d7fa13d7-kube-api-access-wjff8\" (UniqueName: \"kubernetes.io/projected/adff5c77-695a-4300-af20-ee81d7fa13d7-kube-api-access-wjff8\") pod \"adff5c77-695a-4300-af20-ee81d7fa13d7\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") "
Oct 31 19:02:56.011569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.011526    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adff5c77-695a-4300-af20-ee81d7fa13d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adff5c77-695a-4300-af20-ee81d7fa13d7-openshift-service-ca\") pod \"adff5c77-695a-4300-af20-ee81d7fa13d7\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") "
Oct 31 19:02:56.011569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.011550    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-metrics-certs\") pod \"adff5c77-695a-4300-af20-ee81d7fa13d7\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") "
Oct 31 19:02:56.011689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.011589    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-maas-api-tls\") pod \"adff5c77-695a-4300-af20-ee81d7fa13d7\" (UID: \"adff5c77-695a-4300-af20-ee81d7fa13d7\") "
Oct 31 19:02:56.011928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.011905    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adff5c77-695a-4300-af20-ee81d7fa13d7-openshift-service-ca" pod "adff5c77-695a-4300-af20-ee81d7fa13d7" (UID: "adff5c77-695a-4300-af20-ee81d7fa13d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:02:56.012198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.012174    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adff5c77-695a-4300-af20-ee81d7fa13d7-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:56.013853 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.013825    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-maas-api-tls" pod "adff5c77-695a-4300-af20-ee81d7fa13d7" (UID: "adff5c77-695a-4300-af20-ee81d7fa13d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:02:56.014267 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.014245    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-metrics-certs" pod "adff5c77-695a-4300-af20-ee81d7fa13d7" (UID: "adff5c77-695a-4300-af20-ee81d7fa13d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:02:56.014314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.014265    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adff5c77-695a-4300-af20-ee81d7fa13d7-kube-api-access-wjff8" pod "adff5c77-695a-4300-af20-ee81d7fa13d7" (UID: "adff5c77-695a-4300-af20-ee81d7fa13d7"). InnerVolumeSpecName "kube-api-access-wjff8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:56.113022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.112980    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjff8\" (UniqueName: \"kubernetes.io/projected/adff5c77-695a-4300-af20-ee81d7fa13d7-kube-api-access-wjff8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:56.113022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.113014    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:56.113022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.113029    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adff5c77-695a-4300-af20-ee81d7fa13d7-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:56.404363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.404325    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"]
Oct 31 19:02:56.408419 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.408389    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b11e44-788b5cdffc-4q94n"]
Oct 31 19:02:56.910058 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:56.910021    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adff5c77-695a-4300-af20-ee81d7fa13d7" path="/var/lib/kubelet/pods/adff5c77-695a-4300-af20-ee81d7fa13d7/volumes"
Oct 31 19:02:57.358010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.357931    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"]
Oct 31 19:02:57.358445 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.358429    5551 cpu_manager.go:414] "containerMap: removing container" podUID="adff5c77-695a-4300-af20-ee81d7fa13d7" containerName="maas-api"
Oct 31 19:02:57.358523 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.358448    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adff5c77-695a-4300-af20-ee81d7fa13d7" containerName="maas-api"
Oct 31 19:02:57.358523 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.358512    5551 cpu_manager.go:414] "containerMap: removing container" podUID="fe4cf1cb-a637-47b0-81c8-0478c86e5969" containerName="maas-api"
Oct 31 19:02:57.358523 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.358518    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4cf1cb-a637-47b0-81c8-0478c86e5969" containerName="maas-api"
Oct 31 19:02:57.358638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.358608    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4cf1cb-a637-47b0-81c8-0478c86e5969" containerName="maas-api"
Oct 31 19:02:57.358638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.358621    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="adff5c77-695a-4300-af20-ee81d7fa13d7" containerName="maas-api"
Oct 31 19:02:57.363296 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.363279    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.429443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429417    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.429600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429476    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zsvt\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-kube-api-access-4zsvt\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.429600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429576    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.429702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429609    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.429702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429634    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2facf719-8722-451b-8634-bfeeae70f02d-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.429811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429777    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.429904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429888    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.429961 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429918    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2facf719-8722-451b-8634-bfeeae70f02d-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.430017 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.429985    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.461213 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.461188    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"]
Oct 31 19:02:57.530959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.530932    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.530982    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531005    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsvt\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-kube-api-access-4zsvt\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531033    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531055    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531072    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2facf719-8722-451b-8634-bfeeae70f02d-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531451 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531239    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531451 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531309    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531451 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531365    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531451 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531361    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531451 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531411    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2facf719-8722-451b-8634-bfeeae70f02d-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531655    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.531743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.531687    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.532068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.532047    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2facf719-8722-451b-8634-bfeeae70f02d-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.533608 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.533585    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.533853 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.533833    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2facf719-8722-451b-8634-bfeeae70f02d-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.593954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.593919    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsvt\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-kube-api-access-4zsvt\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.594580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.594552    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:57.676889 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:57.676858    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:58.119225 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:58.119194    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"]
Oct 31 19:02:58.120519 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:02:58.120488    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2facf719_8722_451b_8634_bfeeae70f02d.slice/crio-e09aa2ebd98fd5aafbe5b5cda55e2a6ae12fd5ff82afa7ccc22c06fab747cad4 WatchSource:0}: Error finding container e09aa2ebd98fd5aafbe5b5cda55e2a6ae12fd5ff82afa7ccc22c06fab747cad4: Status 404 returned error can't find the container with id e09aa2ebd98fd5aafbe5b5cda55e2a6ae12fd5ff82afa7ccc22c06fab747cad4
Oct 31 19:02:58.122929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:58.122899    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:58.123010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:58.122965    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:58.123010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:58.122992    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:02:59.008314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:59.008277    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" event={"ID":"2facf719-8722-451b-8634-bfeeae70f02d","Type":"ContainerStarted","Data":"525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185"}
Oct 31 19:02:59.008513 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:59.008332    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" event={"ID":"2facf719-8722-451b-8634-bfeeae70f02d","Type":"ContainerStarted","Data":"e09aa2ebd98fd5aafbe5b5cda55e2a6ae12fd5ff82afa7ccc22c06fab747cad4"}
Oct 31 19:02:59.148669 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:59.148619    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" podStartSLOduration=2.148601853 podStartE2EDuration="2.148601853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:59.145767504 +0000 UTC m=+2482.377005806" watchObservedRunningTime="2026-10-31 19:02:59.148601853 +0000 UTC m=+2482.379840144"
Oct 31 19:02:59.680678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:59.680642    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:02:59.681961 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:59.681934    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:02:59.682182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:02:59.682156    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 19:03:00.681482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:00.681105    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:03:00.681482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:00.681176    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 19:03:01.677484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:01.677355    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:03:01.677484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:01.677419    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 19:03:02.678108 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:02.678043    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:03:02.678955 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:02.678916    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 19:03:03.680485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:03.678865    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:03:03.680485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:03.678933    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 19:03:04.181799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:04.181753    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"]
Oct 31 19:03:04.182057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:04.182027    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy" containerID="cri-o://c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208" gracePeriod=30
Oct 31 19:03:04.677786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:04.677746    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:03:04.677966 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:04.677813    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 31 19:03:05.687360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:05.687330    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:03:05.687909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:05.687686    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:03:05.688393 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:05.688370    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:03:07.011484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.009683    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"]
Oct 31 19:03:07.011484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.009984    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6" podUID="28562d13-5ce8-41c9-b134-dcde44413880" containerName="maas-api" containerID="cri-o://0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962" gracePeriod=30
Oct 31 19:03:07.443716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.443687    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 19:03:07.541308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.541216    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls\") pod \"28562d13-5ce8-41c9-b134-dcde44413880\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") "
Oct 31 19:03:07.541541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.541340    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28562d13-5ce8-41c9-b134-dcde44413880-kube-api-access-grbwm\" (UniqueName: \"kubernetes.io/projected/28562d13-5ce8-41c9-b134-dcde44413880-kube-api-access-grbwm\") pod \"28562d13-5ce8-41c9-b134-dcde44413880\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") "
Oct 31 19:03:07.541541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.541383    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-metrics-certs\") pod \"28562d13-5ce8-41c9-b134-dcde44413880\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") "
Oct 31 19:03:07.541541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.541411    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28562d13-5ce8-41c9-b134-dcde44413880-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28562d13-5ce8-41c9-b134-dcde44413880-openshift-service-ca\") pod \"28562d13-5ce8-41c9-b134-dcde44413880\" (UID: \"28562d13-5ce8-41c9-b134-dcde44413880\") "
Oct 31 19:03:07.541820 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.541788    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28562d13-5ce8-41c9-b134-dcde44413880-openshift-service-ca" pod "28562d13-5ce8-41c9-b134-dcde44413880" (UID: "28562d13-5ce8-41c9-b134-dcde44413880"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:07.544126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.544102    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28562d13-5ce8-41c9-b134-dcde44413880-kube-api-access-grbwm" pod "28562d13-5ce8-41c9-b134-dcde44413880" (UID: "28562d13-5ce8-41c9-b134-dcde44413880"). InnerVolumeSpecName "kube-api-access-grbwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:07.544215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.544188    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-metrics-certs" pod "28562d13-5ce8-41c9-b134-dcde44413880" (UID: "28562d13-5ce8-41c9-b134-dcde44413880"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:03:07.544270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.544216    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls" pod "28562d13-5ce8-41c9-b134-dcde44413880" (UID: "28562d13-5ce8-41c9-b134-dcde44413880"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:03:07.643230 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.643158    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:07.643230 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.643189    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbwm\" (UniqueName: \"kubernetes.io/projected/28562d13-5ce8-41c9-b134-dcde44413880-kube-api-access-grbwm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:07.643230 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.643200    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28562d13-5ce8-41c9-b134-dcde44413880-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:07.643230 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:07.643208    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28562d13-5ce8-41c9-b134-dcde44413880-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:08.174273 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.174239    5551 generic.go:356] "Generic (PLEG): container finished" podID="28562d13-5ce8-41c9-b134-dcde44413880" containerID="0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962" exitCode=0
Oct 31 19:03:08.174774 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.174284    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"
Oct 31 19:03:08.174774 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.174334    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6" event={"ID":"28562d13-5ce8-41c9-b134-dcde44413880","Type":"ContainerDied","Data":"0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962"}
Oct 31 19:03:08.174774 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.174397    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6" event={"ID":"28562d13-5ce8-41c9-b134-dcde44413880","Type":"ContainerDied","Data":"1c01473f75cc030e01fda5689ed700d258b27da7542e56ad0b48a59a1cb55ea3"}
Oct 31 19:03:08.174774 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.174432    5551 scope.go:122] "RemoveContainer" containerID="0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962"
Oct 31 19:03:08.185687 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.185671    5551 scope.go:122] "RemoveContainer" containerID="0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962"
Oct 31 19:03:08.185932 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:08.185913    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962\": container with ID starting with 0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962 not found: ID does not exist" containerID="0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962"
Oct 31 19:03:08.185977 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.185940    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962"} err="failed to get container status \"0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962\": rpc error: code = NotFound desc = could not find container \"0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962\": container with ID starting with 0008c90a89a830f032100beedc9b04e0a028a5e3c8ff3b09703e2522c9110962 not found: ID does not exist"
Oct 31 19:03:08.337652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.337618    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"]
Oct 31 19:03:08.387940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.387900    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6"]
Oct 31 19:03:08.908969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:08.908937    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28562d13-5ce8-41c9-b134-dcde44413880" path="/var/lib/kubelet/pods/28562d13-5ce8-41c9-b134-dcde44413880/volumes"
Oct 31 19:03:09.440990 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.440956    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 19:03:09.566731 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.566629    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-data\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.566731 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.566715    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-podinfo\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.566971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.566780    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-token\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-token\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.566971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.566809    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-kube-api-access-8j6cj\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-kube-api-access-8j6cj\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.566971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.566853    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-credential-socket\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.566971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.566892    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-certs\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.566971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.566957    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-envoy\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.566971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.566958    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-data" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:09.567281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.567109    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istiod-ca-cert\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.567281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.567227    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-socket\") pod \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\" (UID: \"bcc92adf-66a4-4b82-8d77-d1d6760af6f3\") "
Oct 31 19:03:09.567281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.567248    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-credential-socket" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:09.567506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.567481    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-certs" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:09.567600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.567526    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istiod-ca-cert" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:09.567856 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.567678    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-socket" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:09.568065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.568044    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:09.568157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.568070    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:09.568157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.568084    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:09.568157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.568100    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:09.568157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.568113    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:09.569633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.569511    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-token" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:09.569633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.569607    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-podinfo" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:03:09.570202 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.570169    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-envoy" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:09.570320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.570280    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-kube-api-access-8j6cj" pod "bcc92adf-66a4-4b82-8d77-d1d6760af6f3" (UID: "bcc92adf-66a4-4b82-8d77-d1d6760af6f3"). InnerVolumeSpecName "kube-api-access-8j6cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:09.669153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.669114    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:09.669153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.669143    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:09.669153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.669155    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:09.669153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:09.669163    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j6cj\" (UniqueName: \"kubernetes.io/projected/bcc92adf-66a4-4b82-8d77-d1d6760af6f3-kube-api-access-8j6cj\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:10.193413 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.193378    5551 generic.go:356] "Generic (PLEG): container finished" podID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerID="c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208" exitCode=0
Oct 31 19:03:10.193610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.193446    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"
Oct 31 19:03:10.193610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.193494    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" event={"ID":"bcc92adf-66a4-4b82-8d77-d1d6760af6f3","Type":"ContainerDied","Data":"c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208"}
Oct 31 19:03:10.193610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.193555    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l" event={"ID":"bcc92adf-66a4-4b82-8d77-d1d6760af6f3","Type":"ContainerDied","Data":"742ef78ba0e8604659268dbfde7478d589c7047d54d70f828868be62b92b5e56"}
Oct 31 19:03:10.193610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.193579    5551 scope.go:122] "RemoveContainer" containerID="c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208"
Oct 31 19:03:10.204808 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.204789    5551 scope.go:122] "RemoveContainer" containerID="6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41"
Oct 31 19:03:10.213651 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.213633    5551 scope.go:122] "RemoveContainer" containerID="c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208"
Oct 31 19:03:10.213917 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:10.213897    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208\": container with ID starting with c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208 not found: ID does not exist" containerID="c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208"
Oct 31 19:03:10.213965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.213925    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208"} err="failed to get container status \"c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208\": rpc error: code = NotFound desc = could not find container \"c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208\": container with ID starting with c461364209699c01945cb051679cdb7aba03af0a64254e1d0d4f31a55fc7e208 not found: ID does not exist"
Oct 31 19:03:10.213965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.213943    5551 scope.go:122] "RemoveContainer" containerID="6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41"
Oct 31 19:03:10.214217 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:10.214189    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41\": container with ID starting with 6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41 not found: ID does not exist" containerID="6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41"
Oct 31 19:03:10.214275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.214228    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41"} err="failed to get container status \"6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41\": rpc error: code = NotFound desc = could not find container \"6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41\": container with ID starting with 6b41898a86001c2e7509287a3c58ea0551d8548e7b5e86165c68e743095d3d41 not found: ID does not exist"
Oct 31 19:03:10.407529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.407486    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"]
Oct 31 19:03:10.413193 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.413171    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b11e44-openshift-default-7bf8d94f69-2m96l"]
Oct 31 19:03:10.438346 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.438310    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-713bf0-7d87cfff4-cxgf6" podUID="28562d13-5ce8-41c9-b134-dcde44413880" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.134:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:03:10.916554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.916519    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" path="/var/lib/kubelet/pods/bcc92adf-66a4-4b82-8d77-d1d6760af6f3/volumes"
Oct 31 19:03:10.985588 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.985558    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"]
Oct 31 19:03:10.986777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:10.986744    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy" containerID="cri-o://f8e3a49d6ad6ec06fbc48c68d38570c87b5cd0f5f3904449673654ef4131fa7e" gracePeriod=30
Oct 31 19:03:12.179853 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.179815    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"]
Oct 31 19:03:12.180674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180651    5551 cpu_manager.go:414] "containerMap: removing container" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy"
Oct 31 19:03:12.180674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180675    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy"
Oct 31 19:03:12.180863 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180693    5551 cpu_manager.go:414] "containerMap: removing container" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy"
Oct 31 19:03:12.180863 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180701    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy"
Oct 31 19:03:12.180863 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180733    5551 cpu_manager.go:414] "containerMap: removing container" podUID="28562d13-5ce8-41c9-b134-dcde44413880" containerName="maas-api"
Oct 31 19:03:12.180863 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180741    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28562d13-5ce8-41c9-b134-dcde44413880" containerName="maas-api"
Oct 31 19:03:12.181071 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180866    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy"
Oct 31 19:03:12.181071 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180879    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc92adf-66a4-4b82-8d77-d1d6760af6f3" containerName="istio-proxy"
Oct 31 19:03:12.181071 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.180891    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="28562d13-5ce8-41c9-b134-dcde44413880" containerName="maas-api"
Oct 31 19:03:12.185849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.185825    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223046    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-certs\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223098    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-credential-socket\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223120    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-data\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223151    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6mp\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-kube-api-access-xw6mp\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223230    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-socket\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223251    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-token\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223313    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa5d1dd-d82d-4484-acfb-c936f2088799-istiod-ca-cert\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223358    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-envoy\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.227483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.223445    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-podinfo\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.270303 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.270257    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"]
Oct 31 19:03:12.314547 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.314519    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1ef01f-openshift-default-dockercfg-zv56g\""
Oct 31 19:03:12.324312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324283    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-socket\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324314    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-token\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324469    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa5d1dd-d82d-4484-acfb-c936f2088799-istiod-ca-cert\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324513    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-envoy\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324634 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324617    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-podinfo\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324704    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-certs\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324729    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-credential-socket\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324744    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-data\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324769    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6mp\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-kube-api-access-xw6mp\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.324936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.324700    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-socket\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.325452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.325233    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-certs\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.325452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.325316    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa5d1dd-d82d-4484-acfb-c936f2088799-istiod-ca-cert\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.325452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.325318    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-data\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.325737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.325574    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-credential-socket\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.327024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.327004    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-envoy\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.327315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.327297    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-podinfo\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.381739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.381700    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-token\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.381918 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.381826    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6mp\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-kube-api-access-xw6mp\") pod \"e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") " pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.501841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.501744    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:12.938904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.938880    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"]
Oct 31 19:03:12.944192 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:03:12.944137    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa5d1dd_d82d_4484_acfb_c936f2088799.slice/crio-b286b588d701853449ed281194af19abed4c425487c4bfc7795536e48e1f7474 WatchSource:0}: Error finding container b286b588d701853449ed281194af19abed4c425487c4bfc7795536e48e1f7474: Status 404 returned error can't find the container with id b286b588d701853449ed281194af19abed4c425487c4bfc7795536e48e1f7474
Oct 31 19:03:12.946027 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.945993    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:12.946129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.946067    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:12.946129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:12.946110    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:13.223413 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:13.223324    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" event={"ID":"faa5d1dd-d82d-4484-acfb-c936f2088799","Type":"ContainerStarted","Data":"03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490"}
Oct 31 19:03:13.223413 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:13.223389    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" event={"ID":"faa5d1dd-d82d-4484-acfb-c936f2088799","Type":"ContainerStarted","Data":"b286b588d701853449ed281194af19abed4c425487c4bfc7795536e48e1f7474"}
Oct 31 19:03:13.340869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:13.340809    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" podStartSLOduration=1.340791302 podStartE2EDuration="1.340791302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:13.338287393 +0000 UTC m=+2496.569525684" watchObservedRunningTime="2026-10-31 19:03:13.340791302 +0000 UTC m=+2496.572029593"
Oct 31 19:03:13.502146 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:13.502058    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:14.509073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:14.509044    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:15.237506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:15.237481    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:15.238563 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:15.238543    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:03:16.246107 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.246073    5551 generic.go:356] "Generic (PLEG): container finished" podID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerID="f8e3a49d6ad6ec06fbc48c68d38570c87b5cd0f5f3904449673654ef4131fa7e" exitCode=0
Oct 31 19:03:16.246430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.246111    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" event={"ID":"320c4562-4bfc-4db1-8839-068ae02a7dc9","Type":"ContainerDied","Data":"f8e3a49d6ad6ec06fbc48c68d38570c87b5cd0f5f3904449673654ef4131fa7e"}
Oct 31 19:03:16.246430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.246166    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6" event={"ID":"320c4562-4bfc-4db1-8839-068ae02a7dc9","Type":"ContainerDied","Data":"36e8857f843fa47db6bacffc4d144db79dbd5e3e4b116c7c42ca93396beb747a"}
Oct 31 19:03:16.246430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.246186    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e8857f843fa47db6bacffc4d144db79dbd5e3e4b116c7c42ca93396beb747a"
Oct 31 19:03:16.246430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.246203    5551 scope.go:122] "RemoveContainer" containerID="8de1d6332ee4dd0b1b1e2ff8b0cbcbcb04ea109681639b861e0f928c8218217e"
Oct 31 19:03:16.251755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.251734    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 19:03:16.273760 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.273703    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320c4562-4bfc-4db1-8839-068ae02a7dc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320c4562-4bfc-4db1-8839-068ae02a7dc9-istiod-ca-cert\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.273864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.273773    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-token\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-token\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.273864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.273829    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-kube-api-access-cx7jk\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-kube-api-access-cx7jk\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.273864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.273859    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-data\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.274023 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.273993    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-credential-socket\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.274083 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274032    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-certs\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.274083 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274036    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320c4562-4bfc-4db1-8839-068ae02a7dc9-istiod-ca-cert" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:16.274195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274109    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-podinfo\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.274195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274142    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-data" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:16.274195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274165    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-socket\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.274347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274201    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-envoy\") pod \"320c4562-4bfc-4db1-8839-068ae02a7dc9\" (UID: \"320c4562-4bfc-4db1-8839-068ae02a7dc9\") "
Oct 31 19:03:16.274347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274231    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-credential-socket" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:16.274444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274329    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-certs" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:16.274444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.274431    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-socket" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:16.275084 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.275061    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.275203 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.275089    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.275203 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.275106    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.275203 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.275121    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.275203 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.275134    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320c4562-4bfc-4db1-8839-068ae02a7dc9-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.276255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.276230    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-token" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:16.276545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.276514    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-envoy" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:16.276645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.276625    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-kube-api-access-cx7jk" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "kube-api-access-cx7jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:16.276806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.276742    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-podinfo" pod "320c4562-4bfc-4db1-8839-068ae02a7dc9" (UID: "320c4562-4bfc-4db1-8839-068ae02a7dc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:03:16.376003 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.375965    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.376003 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.375998    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7jk\" (UniqueName: \"kubernetes.io/projected/320c4562-4bfc-4db1-8839-068ae02a7dc9-kube-api-access-cx7jk\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.376003 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.376010    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.376230 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:16.376019    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320c4562-4bfc-4db1-8839-068ae02a7dc9-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:16.604771 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:16.604686    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:03:17.254553 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:17.254450    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"
Oct 31 19:03:17.486383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:17.486353    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"]
Oct 31 19:03:17.491011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:17.490986    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-713bf0-openshift-default-597fb86869-5ncr6"]
Oct 31 19:03:18.909815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:18.909769    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" path="/var/lib/kubelet/pods/320c4562-4bfc-4db1-8839-068ae02a7dc9/volumes"
Oct 31 19:03:23.715880 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:23.715847    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"]
Oct 31 19:03:23.716324 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:23.716047    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q" podUID="7477fcef-3dfe-415c-b8f7-80c2cc3cd602" containerName="maas-api" containerID="cri-o://bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d" gracePeriod=30
Oct 31 19:03:23.980667 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:23.980593    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:03:24.063102 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.063073    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-metrics-certs\") pod \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") "
Oct 31 19:03:24.063301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.063117    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-openshift-service-ca\") pod \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") "
Oct 31 19:03:24.063301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.063186    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls\") pod \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") "
Oct 31 19:03:24.063301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.063239    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-kube-api-access-pzs2g\" (UniqueName: \"kubernetes.io/projected/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-kube-api-access-pzs2g\") pod \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\" (UID: \"7477fcef-3dfe-415c-b8f7-80c2cc3cd602\") "
Oct 31 19:03:24.063818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.063616    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-openshift-service-ca" pod "7477fcef-3dfe-415c-b8f7-80c2cc3cd602" (UID: "7477fcef-3dfe-415c-b8f7-80c2cc3cd602"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:24.063818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.063794    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:24.065671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.065644    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-metrics-certs" pod "7477fcef-3dfe-415c-b8f7-80c2cc3cd602" (UID: "7477fcef-3dfe-415c-b8f7-80c2cc3cd602"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:03:24.065962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.065942    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls" pod "7477fcef-3dfe-415c-b8f7-80c2cc3cd602" (UID: "7477fcef-3dfe-415c-b8f7-80c2cc3cd602"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:03:24.066348 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.066324    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-kube-api-access-pzs2g" pod "7477fcef-3dfe-415c-b8f7-80c2cc3cd602" (UID: "7477fcef-3dfe-415c-b8f7-80c2cc3cd602"). InnerVolumeSpecName "kube-api-access-pzs2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:24.165605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.165572    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:24.165605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.165606    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:24.165809 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.165623    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzs2g\" (UniqueName: \"kubernetes.io/projected/7477fcef-3dfe-415c-b8f7-80c2cc3cd602-kube-api-access-pzs2g\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:24.301200 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.301121    5551 generic.go:356] "Generic (PLEG): container finished" podID="7477fcef-3dfe-415c-b8f7-80c2cc3cd602" containerID="bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d" exitCode=0
Oct 31 19:03:24.301200 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.301177    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"
Oct 31 19:03:24.301414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.301207    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q" event={"ID":"7477fcef-3dfe-415c-b8f7-80c2cc3cd602","Type":"ContainerDied","Data":"bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d"}
Oct 31 19:03:24.301414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.301264    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q" event={"ID":"7477fcef-3dfe-415c-b8f7-80c2cc3cd602","Type":"ContainerDied","Data":"29ce0c4114ca0054bc24a78d9e74939cfcc2831bfbf9bd59b5294573e21f81a2"}
Oct 31 19:03:24.301414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.301287    5551 scope.go:122] "RemoveContainer" containerID="bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d"
Oct 31 19:03:24.312166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.312149    5551 scope.go:122] "RemoveContainer" containerID="bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d"
Oct 31 19:03:24.312438 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:24.312419    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d\": container with ID starting with bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d not found: ID does not exist" containerID="bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d"
Oct 31 19:03:24.312532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.312447    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d"} err="failed to get container status \"bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d\": rpc error: code = NotFound desc = could not find container \"bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d\": container with ID starting with bccf22f8703d24247a537f5cacd2fd797e1a5acdf75224d2f285891df8fba92d not found: ID does not exist"
Oct 31 19:03:24.463998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.463958    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"]
Oct 31 19:03:24.468677 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.468628    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4c58c5-7b8658db78-tdh2q"]
Oct 31 19:03:24.908239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:24.908201    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7477fcef-3dfe-415c-b8f7-80c2cc3cd602" path="/var/lib/kubelet/pods/7477fcef-3dfe-415c-b8f7-80c2cc3cd602/volumes"
Oct 31 19:03:34.332825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:34.332780    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"]
Oct 31 19:03:34.333278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:34.333015    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy" containerID="cri-o://15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449" gracePeriod=30
Oct 31 19:03:34.335520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:34.335495    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"]
Oct 31 19:03:34.335754 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:34.335708    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy" containerID="cri-o://848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b" gracePeriod=30
Oct 31 19:03:34.337977 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:34.337951    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"]
Oct 31 19:03:34.338133 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:34.338113    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy" containerID="cri-o://5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d" gracePeriod=30
Oct 31 19:03:35.288633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:35.288601    5551 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:03:35.288793 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:35.288651    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:03:36.359668 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.359619    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"]
Oct 31 19:03:36.360403 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360382    5551 cpu_manager.go:414] "containerMap: removing container" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy"
Oct 31 19:03:36.360476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360406    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy"
Oct 31 19:03:36.360476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360426    5551 cpu_manager.go:414] "containerMap: removing container" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy"
Oct 31 19:03:36.360476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360435    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy"
Oct 31 19:03:36.360590 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360514    5551 cpu_manager.go:414] "containerMap: removing container" podUID="7477fcef-3dfe-415c-b8f7-80c2cc3cd602" containerName="maas-api"
Oct 31 19:03:36.360590 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360523    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7477fcef-3dfe-415c-b8f7-80c2cc3cd602" containerName="maas-api"
Oct 31 19:03:36.360678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360665    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="7477fcef-3dfe-415c-b8f7-80c2cc3cd602" containerName="maas-api"
Oct 31 19:03:36.360715 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360687    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy"
Oct 31 19:03:36.360715 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.360696    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="320c4562-4bfc-4db1-8839-068ae02a7dc9" containerName="istio-proxy"
Oct 31 19:03:36.364696 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.364678    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.417921 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.417886    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a2c587-openshift-default-dockercfg-667vh\""
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435546    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-envoy\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435597    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-credential-socket\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435646    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-data\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435722    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-socket\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435805    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istiod-ca-cert\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435826    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-podinfo\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435860    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-certs\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435894    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqvl\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-kube-api-access-zwqvl\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.436791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.435916    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-token\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.470325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.470291    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"]
Oct 31 19:03:36.536727 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536695    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-socket\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.536913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536775    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istiod-ca-cert\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.536913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536796    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-podinfo\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.536913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536822    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-certs\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.536913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536845    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqvl\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-kube-api-access-zwqvl\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.536913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536862    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-token\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.536913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536896    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-envoy\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.537255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536928    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-credential-socket\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.537255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.536981    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-data\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.537255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.537137    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-socket\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.537255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.537179    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-certs\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.537503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.537327    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-data\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.537503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.537432    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-credential-socket\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.537825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.537803    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istiod-ca-cert\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.539237 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.539217    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-envoy\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.539447 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.539429    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-podinfo\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.594838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.594813    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqvl\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-kube-api-access-zwqvl\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.595003 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.594920    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-token\") pod \"e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") " pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:36.678603 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:36.678568    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:37.105967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:37.105936    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"]
Oct 31 19:03:37.109959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:37.109922    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:37.110053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:37.109999    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:37.110053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:37.110027    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:37.390024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:37.389987    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" event={"ID":"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1","Type":"ContainerStarted","Data":"a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5"}
Oct 31 19:03:37.390406 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:37.390053    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" event={"ID":"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1","Type":"ContainerStarted","Data":"f6e4dcabd0d93b96cc58865d619a9fea504292da8765b6d6883029032b91a200"}
Oct 31 19:03:37.505325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:37.505261    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" podStartSLOduration=1.5052410859999998 podStartE2EDuration="1.505241086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:37.503178301 +0000 UTC m=+2520.734416593" watchObservedRunningTime="2026-10-31 19:03:37.505241086 +0000 UTC m=+2520.736479376"
Oct 31 19:03:37.679810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:37.679725    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:38.686301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:38.686272    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:39.403006 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.402974    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:39.403881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.403860    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:03:39.646529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.646069    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:03:39.652590 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.652567    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.673655    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-token\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-token\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.673732    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-certs\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.673775    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-data\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.673806    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-credential-socket\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.673837    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-kube-api-access-whnk2\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-kube-api-access-whnk2\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.673906    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/762f8ee8-681f-495c-b902-75a8d1214edf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762f8ee8-681f-495c-b902-75a8d1214edf-istio-podinfo\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.673938    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-socket\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.673987    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-kube-api-access-l6mpv\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-kube-api-access-l6mpv\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674033    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-podinfo\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674054    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-istio-token\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-istio-token\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674099    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-credential-socket\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674135    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-socket\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674162    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762f8ee8-681f-495c-b902-75a8d1214edf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762f8ee8-681f-495c-b902-75a8d1214edf-istiod-ca-cert\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674206    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-data\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674235    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-certs\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674264    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-envoy\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674287    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-envoy\") pod \"762f8ee8-681f-495c-b902-75a8d1214edf\" (UID: \"762f8ee8-681f-495c-b902-75a8d1214edf\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.674322    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530506f9-6a80-4249-8c5d-890be1bb26c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530506f9-6a80-4249-8c5d-890be1bb26c5-istiod-ca-cert\") pod \"530506f9-6a80-4249-8c5d-890be1bb26c5\" (UID: \"530506f9-6a80-4249-8c5d-890be1bb26c5\") "
Oct 31 19:03:39.676499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.675088    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530506f9-6a80-4249-8c5d-890be1bb26c5-istiod-ca-cert" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.677047    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-credential-socket" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.677312    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-certs" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.677937    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-socket" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.677920    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-data" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.678161    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-socket" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.678210    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-data" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.679030    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-certs" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.679078    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-credential-socket" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.681160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.679727    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762f8ee8-681f-495c-b902-75a8d1214edf-istiod-ca-cert" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:39.682383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.682354    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-kube-api-access-l6mpv" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "kube-api-access-l6mpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.682599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.682575    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-kube-api-access-whnk2" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "kube-api-access-whnk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.682706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.682602    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-token" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.682773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.682711    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-istio-token" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.683566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.683535    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-podinfo" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:03:39.683660 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.683572    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-envoy" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.684093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.684057    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-envoy" pod "530506f9-6a80-4249-8c5d-890be1bb26c5" (UID: "530506f9-6a80-4249-8c5d-890be1bb26c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.684445 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.684430    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/762f8ee8-681f-495c-b902-75a8d1214edf-istio-podinfo" pod "762f8ee8-681f-495c-b902-75a8d1214edf" (UID: "762f8ee8-681f-495c-b902-75a8d1214edf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:03:39.695001 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.694979    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:03:39.775751 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.775723    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-data\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.775929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.775767    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32598a29-4328-48fd-98dc-f896ef2d96ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32598a29-4328-48fd-98dc-f896ef2d96ed-istiod-ca-cert\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.775929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.775806    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-podinfo\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.775929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.775831    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-certs\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.775929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.775861    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-socket\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.776157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776031    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-data" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.776157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776092    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-kube-api-access-qw7nz\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-kube-api-access-qw7nz\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.776274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776189    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-certs" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.776274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776208    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-envoy\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.776388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776325    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-credential-socket\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.776443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776423    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-token\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-token\") pod \"32598a29-4328-48fd-98dc-f896ef2d96ed\" (UID: \"32598a29-4328-48fd-98dc-f896ef2d96ed\") "
Oct 31 19:03:39.776768 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776747    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32598a29-4328-48fd-98dc-f896ef2d96ed-istiod-ca-cert" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:39.776891 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776868    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-socket" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.776973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776954    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6mpv\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-kube-api-access-l6mpv\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.776984    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777000    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762f8ee8-681f-495c-b902-75a8d1214edf-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777014    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777029    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777044    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762f8ee8-681f-495c-b902-75a8d1214edf-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777057    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777067    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-credential-socket" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777073    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32598a29-4328-48fd-98dc-f896ef2d96ed-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777108    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777125    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777141    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777159    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777173    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777190    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530506f9-6a80-4249-8c5d-890be1bb26c5-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.777199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777204    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.778062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777219    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.778062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777232    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.778062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777247    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530506f9-6a80-4249-8c5d-890be1bb26c5-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.778062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777262    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whnk2\" (UniqueName: \"kubernetes.io/projected/530506f9-6a80-4249-8c5d-890be1bb26c5-kube-api-access-whnk2\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.778062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777277    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762f8ee8-681f-495c-b902-75a8d1214edf-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.778062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.777292    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762f8ee8-681f-495c-b902-75a8d1214edf-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.778988 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.778944    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-podinfo" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:03:39.779363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.779335    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-kube-api-access-qw7nz" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "kube-api-access-qw7nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.779524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.779381    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-token" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.779524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.779514    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-envoy" pod "32598a29-4328-48fd-98dc-f896ef2d96ed" (UID: "32598a29-4328-48fd-98dc-f896ef2d96ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.879956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.878966    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.879956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.878997    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.879956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.879026    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw7nz\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-kube-api-access-qw7nz\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.879956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.879043    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.879956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.879058    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32598a29-4328-48fd-98dc-f896ef2d96ed-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.879956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:39.879070    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32598a29-4328-48fd-98dc-f896ef2d96ed-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:40.411487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.411435    5551 generic.go:356] "Generic (PLEG): container finished" podID="762f8ee8-681f-495c-b902-75a8d1214edf" containerID="15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449" exitCode=0
Oct 31 19:03:40.411713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.411517    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"
Oct 31 19:03:40.411713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.411516    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" event={"ID":"762f8ee8-681f-495c-b902-75a8d1214edf","Type":"ContainerDied","Data":"15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449"}
Oct 31 19:03:40.411713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.411619    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv" event={"ID":"762f8ee8-681f-495c-b902-75a8d1214edf","Type":"ContainerDied","Data":"f12701d6dbeff98717dfcf948a45cf10dea3456609ee834fedbe853e5eb86928"}
Oct 31 19:03:40.411713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.411658    5551 scope.go:122] "RemoveContainer" containerID="15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449"
Oct 31 19:03:40.413370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.413332    5551 generic.go:356] "Generic (PLEG): container finished" podID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerID="5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d" exitCode=0
Oct 31 19:03:40.413530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.413412    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"
Oct 31 19:03:40.413530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.413431    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" event={"ID":"530506f9-6a80-4249-8c5d-890be1bb26c5","Type":"ContainerDied","Data":"5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d"}
Oct 31 19:03:40.413530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.413524    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw" event={"ID":"530506f9-6a80-4249-8c5d-890be1bb26c5","Type":"ContainerDied","Data":"f2984536d63cf18546cb98fea03997c60d586f12487aec276fe3b4a6616c3e6c"}
Oct 31 19:03:40.415405 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.415375    5551 generic.go:356] "Generic (PLEG): container finished" podID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerID="848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b" exitCode=0
Oct 31 19:03:40.415550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.415470    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"
Oct 31 19:03:40.415550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.415489    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" event={"ID":"32598a29-4328-48fd-98dc-f896ef2d96ed","Type":"ContainerDied","Data":"848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b"}
Oct 31 19:03:40.415640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.415571    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5" event={"ID":"32598a29-4328-48fd-98dc-f896ef2d96ed","Type":"ContainerDied","Data":"6de15151fdaa67a08820af61b27721e882ebea4896d968808181ac6892d94146"}
Oct 31 19:03:40.424295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.424271    5551 scope.go:122] "RemoveContainer" containerID="38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470"
Oct 31 19:03:40.434053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.434029    5551 scope.go:122] "RemoveContainer" containerID="15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449"
Oct 31 19:03:40.434321 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:40.434303    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449\": container with ID starting with 15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449 not found: ID does not exist" containerID="15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449"
Oct 31 19:03:40.434383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.434327    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449"} err="failed to get container status \"15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449\": rpc error: code = NotFound desc = could not find container \"15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449\": container with ID starting with 15e822d7de7fb1c19cc17ef524aa5a49f6e10e54d908e4294551f4fc8853b449 not found: ID does not exist"
Oct 31 19:03:40.434383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.434343    5551 scope.go:122] "RemoveContainer" containerID="38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470"
Oct 31 19:03:40.434585 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:40.434567    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470\": container with ID starting with 38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470 not found: ID does not exist" containerID="38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470"
Oct 31 19:03:40.434636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.434593    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470"} err="failed to get container status \"38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470\": rpc error: code = NotFound desc = could not find container \"38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470\": container with ID starting with 38f9e86cd6ef481caaef9cc77be1fef1e81bde0fbdde265e1f96e4f890bfc470 not found: ID does not exist"
Oct 31 19:03:40.434636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.434611    5551 scope.go:122] "RemoveContainer" containerID="5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d"
Oct 31 19:03:40.444210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.444191    5551 scope.go:122] "RemoveContainer" containerID="5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d"
Oct 31 19:03:40.444489 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:40.444445    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d\": container with ID starting with 5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d not found: ID does not exist" containerID="5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d"
Oct 31 19:03:40.444537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.444498    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d"} err="failed to get container status \"5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d\": rpc error: code = NotFound desc = could not find container \"5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d\": container with ID starting with 5a9671ffe520defff298132627c17e29b47fbe0c7ca5f5ef3270bde50954c87d not found: ID does not exist"
Oct 31 19:03:40.444537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.444518    5551 scope.go:122] "RemoveContainer" containerID="848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b"
Oct 31 19:03:40.452905 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.452888    5551 scope.go:122] "RemoveContainer" containerID="848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b"
Oct 31 19:03:40.453162 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:40.453145    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b\": container with ID starting with 848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b not found: ID does not exist" containerID="848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b"
Oct 31 19:03:40.453205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.453169    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b"} err="failed to get container status \"848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b\": rpc error: code = NotFound desc = could not find container \"848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b\": container with ID starting with 848a0acc66818884b79fdfdeb1951f75fabfc038aff8e53df1a084c6e2c63b9b not found: ID does not exist"
Oct 31 19:03:40.575497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.575447    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"]
Oct 31 19:03:40.625128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.625101    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-6jcsv"]
Oct 31 19:03:40.685276 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.685246    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"]
Oct 31 19:03:40.691397 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.691369    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-khqlw"]
Oct 31 19:03:40.804812 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.804774    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"]
Oct 31 19:03:40.809517 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.809482    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-fdvt5"]
Oct 31 19:03:40.909966 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.909892    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" path="/var/lib/kubelet/pods/32598a29-4328-48fd-98dc-f896ef2d96ed/volumes"
Oct 31 19:03:40.910376 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.910361    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" path="/var/lib/kubelet/pods/530506f9-6a80-4249-8c5d-890be1bb26c5/volumes"
Oct 31 19:03:40.910783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:40.910770    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" path="/var/lib/kubelet/pods/762f8ee8-681f-495c-b902-75a8d1214edf/volumes"
Oct 31 19:03:42.684812 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.684718    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"]
Oct 31 19:03:42.685501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685482    5551 cpu_manager.go:414] "containerMap: removing container" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy"
Oct 31 19:03:42.685593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685503    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy"
Oct 31 19:03:42.685593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685523    5551 cpu_manager.go:414] "containerMap: removing container" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy"
Oct 31 19:03:42.685593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685534    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy"
Oct 31 19:03:42.685593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685560    5551 cpu_manager.go:414] "containerMap: removing container" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy"
Oct 31 19:03:42.685593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685568    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy"
Oct 31 19:03:42.685847 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685716    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy"
Oct 31 19:03:42.685847 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685734    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="32598a29-4328-48fd-98dc-f896ef2d96ed" containerName="istio-proxy"
Oct 31 19:03:42.685847 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.685746    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="530506f9-6a80-4249-8c5d-890be1bb26c5" containerName="istio-proxy"
Oct 31 19:03:42.691335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.691309    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.743917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.743883    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e4a2c-openshift-default-dockercfg-7jr6n\""
Oct 31 19:03:42.798802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.798769    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"]
Oct 31 19:03:42.819077 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819037    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.819435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819401    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.819623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819516    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.819623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819557    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.819704 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819662    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2xf\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-kube-api-access-2b2xf\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.819789 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819762    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.819895 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819876    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.819945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819929    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.819997 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.819982    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.921539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921508    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.921539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921546    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.921791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921583    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2xf\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-kube-api-access-2b2xf\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.921791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921621    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.921791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921689    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.921791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921716    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.921791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921755    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.921791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921781    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.922073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921816    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.922073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.921959    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.922073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.922031    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.922338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.922314    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.922419 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.922337    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.922419 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.922360    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.924216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.924192    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.924326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.924263    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.978497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.978410    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:42.978704 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:42.978680    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2xf\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-kube-api-access-2b2xf\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:43.005258 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:43.005236    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:43.435061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:43.435036    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"]
Oct 31 19:03:43.436159 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:03:43.436126    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0946f9f5_25ab_4fc5_848f_1fc94017cdcf.slice/crio-49a075b3e2bd6cd3c87d1a40c2c4be4bd82c360b44af37271f28a3b68a8d21d8 WatchSource:0}: Error finding container 49a075b3e2bd6cd3c87d1a40c2c4be4bd82c360b44af37271f28a3b68a8d21d8: Status 404 returned error can't find the container with id 49a075b3e2bd6cd3c87d1a40c2c4be4bd82c360b44af37271f28a3b68a8d21d8
Oct 31 19:03:43.438997 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:43.438966    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:43.439079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:43.439034    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:43.439079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:43.439063    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:43.444777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:43.444755    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" event={"ID":"0946f9f5-25ab-4fc5-848f-1fc94017cdcf","Type":"ContainerStarted","Data":"49a075b3e2bd6cd3c87d1a40c2c4be4bd82c360b44af37271f28a3b68a8d21d8"}
Oct 31 19:03:44.466482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:44.465125    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" event={"ID":"0946f9f5-25ab-4fc5-848f-1fc94017cdcf","Type":"ContainerStarted","Data":"2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731"}
Oct 31 19:03:44.646839 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:44.646787    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podStartSLOduration=2.646772768 podStartE2EDuration="2.646772768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:44.644843947 +0000 UTC m=+2527.876082236" watchObservedRunningTime="2026-10-31 19:03:44.646772768 +0000 UTC m=+2527.878011123"
Oct 31 19:03:45.006479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.006412    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:45.007705 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.007679    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 19:03:45.007850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.007743    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 19:03:45.518430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.518399    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"]
Oct 31 19:03:45.518981 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.518965    5551 cpu_manager.go:414] "containerMap: removing container" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy"
Oct 31 19:03:45.519026 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.518983    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy"
Oct 31 19:03:45.519112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.519102    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="762f8ee8-681f-495c-b902-75a8d1214edf" containerName="istio-proxy"
Oct 31 19:03:45.522721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.522692    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:45.585490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.583721    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a2c587\""
Oct 31 19:03:45.585490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.583856    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a2c587-dockercfg-52fkj\""
Oct 31 19:03:45.610857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.597560    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"]
Oct 31 19:03:45.686772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.675190    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988\" (UID: \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:45.686772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.675382    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvhw\" (UniqueName: \"kubernetes.io/projected/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-kube-api-access-6xvhw\") pod \"payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988\" (UID: \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:45.785364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.783810    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988\" (UID: \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:45.785364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.783963    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvhw\" (UniqueName: \"kubernetes.io/projected/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-kube-api-access-6xvhw\") pod \"payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988\" (UID: \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:45.785364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.785235    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988\" (UID: \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:45.840783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:45.840749    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvhw\" (UniqueName: \"kubernetes.io/projected/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-kube-api-access-6xvhw\") pod \"payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988\" (UID: \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:46.005644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:46.005610    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 19:03:46.005810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:46.005671    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 19:03:46.136859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:46.136829    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:46.546878 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:46.546854    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"]
Oct 31 19:03:46.549325 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:03:46.549296    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a27c086_2d6e_4e13_ae84_fe5327ab8cb7.slice/crio-ebc63b71883be9b8fc2447b4551e6ff09323368935f5d6c0106ec2e6bfdab5e0 WatchSource:0}: Error finding container ebc63b71883be9b8fc2447b4551e6ff09323368935f5d6c0106ec2e6bfdab5e0: Status 404 returned error can't find the container with id ebc63b71883be9b8fc2447b4551e6ff09323368935f5d6c0106ec2e6bfdab5e0
Oct 31 19:03:46.621162 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:46.621132    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:03:47.005636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:47.005601    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 19:03:47.005794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:47.005650    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 19:03:47.503887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:47.503851    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988" event={"ID":"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7","Type":"ContainerStarted","Data":"1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb"}
Oct 31 19:03:47.504074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:47.503907    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988" event={"ID":"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7","Type":"ContainerStarted","Data":"ebc63b71883be9b8fc2447b4551e6ff09323368935f5d6c0106ec2e6bfdab5e0"}
Oct 31 19:03:47.504074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:47.503955    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:47.640896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:47.640834    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988" podStartSLOduration=2.640817724 podStartE2EDuration="2.640817724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:47.638641932 +0000 UTC m=+2530.869880219" watchObservedRunningTime="2026-10-31 19:03:47.640817724 +0000 UTC m=+2530.872056075"
Oct 31 19:03:48.005713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:48.005679    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 19:03:48.005912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:48.005731    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 19:03:49.006354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:49.006319    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 19:03:49.006763 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:49.006372    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 19:03:50.005618 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.005586    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 19:03:50.005942 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.005640    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 19:03:50.399723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.399693    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"]
Oct 31 19:03:50.403834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.403810    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.505697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.505659    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1ef01f-serving-cert\""
Oct 31 19:03:50.510327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.510292    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"]
Oct 31 19:03:50.548836 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.548805    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.549015 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.548851    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-metrics-certs\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.549015 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.548878    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdfr\" (UniqueName: \"kubernetes.io/projected/ca526226-de23-4bd9-b474-162919d2af8e-kube-api-access-jpdfr\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.549015 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.548939    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca526226-de23-4bd9-b474-162919d2af8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.650358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.650262    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.655842 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:50.652618    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-serving-cert: secret "maas-api-e2e-praxis-b-1ef01f-serving-cert" not found
Oct 31 19:03:50.656167 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:03:50.656150    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls podName:ca526226-de23-4bd9-b474-162919d2af8e nodeName:}" failed. No retries permitted until 2026-10-31 19:03:51.156106341 +0000 UTC m=+2534.387344627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls") pod "maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc" (UID: "ca526226-de23-4bd9-b474-162919d2af8e") : secret "maas-api-e2e-praxis-b-1ef01f-serving-cert" not found
Oct 31 19:03:50.657817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.654542    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-metrics-certs\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.658033 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.658004    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdfr\" (UniqueName: \"kubernetes.io/projected/ca526226-de23-4bd9-b474-162919d2af8e-kube-api-access-jpdfr\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.658252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.658233    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca526226-de23-4bd9-b474-162919d2af8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.661679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.661658    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca526226-de23-4bd9-b474-162919d2af8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.672321 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.672271    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-metrics-certs\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:50.715299 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:50.715272    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdfr\" (UniqueName: \"kubernetes.io/projected/ca526226-de23-4bd9-b474-162919d2af8e-kube-api-access-jpdfr\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:51.165089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:51.165049    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:51.167877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:51.167848    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:51.319039 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:51.319007    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:51.722786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:51.718407    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"]
Oct 31 19:03:52.046662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:52.046575    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:03:52.046662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:52.046637    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:03:52.167125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:52.167094    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:52.167514 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:52.167491    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:52.168217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:52.168197    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:03:52.540182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:52.540144    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc" event={"ID":"ca526226-de23-4bd9-b474-162919d2af8e","Type":"ContainerStarted","Data":"eeb47bc96bc22c246d500e4a1f32534eb01b8d8932c6961a63d34e8cf2848b02"}
Oct 31 19:03:53.548800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:53.548757    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc" event={"ID":"ca526226-de23-4bd9-b474-162919d2af8e","Type":"ContainerStarted","Data":"fb305087d2a2f638ccf83e2696dd9a8fc9166ae028546c454ae0f730d9726ecb"}
Oct 31 19:03:53.665145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:53.665083    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc" podStartSLOduration=2.810139847 podStartE2EDuration="3.665064276s" totalImagesPullingTime="854.924429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:53.662423981 +0000 UTC m=+2536.893662272" watchObservedRunningTime="2026-10-31 19:03:53.665064276 +0000 UTC m=+2536.896302567"
Oct 31 19:03:54.557264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:54.557223    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:03:57.810126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.810087    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"]
Oct 31 19:03:57.814153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.814134    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.842543 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.842511    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.842671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.842549    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.842741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.842683    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.842799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.842782    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.842857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.842838    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bwm\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-kube-api-access-b5bwm\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.842922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.842910    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.842959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.842947    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.843008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.842971    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.843100 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.843083    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.859773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.859731    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"]
Oct 31 19:03:57.863968 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.863941    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.915380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.915353    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"]
Oct 31 19:03:57.944421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944395    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.944559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944433    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl45t\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-kube-api-access-dl45t\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.944559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944453    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.944652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944634    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.944690 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944668    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.944740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944687    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.944740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944706    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.944740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944725    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.944886 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944839    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.944886 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944873    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.944994 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944842    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.944994 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.944918    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.945122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945021    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.945122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945027    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.945122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945078    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.945122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945113    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.945379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945149    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.945379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945178    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:57.945379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945179    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.945379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945296    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bwm\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-kube-api-access-b5bwm\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.945379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945338    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.945624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945399    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.945928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.945911    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.946879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.946854    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.947212 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.947194    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:57.965612 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:57.965585    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"]
Oct 31 19:03:58.001326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.001303    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:58.001692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.001675    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bwm\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-kube-api-access-b5bwm\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:58.046598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046565    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.046790 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046606    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.046790 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046661    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.046790 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046694    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.046790 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046771    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046811    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046844    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046872    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.046988    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl45t\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-kube-api-access-dl45t\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047223 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.047019    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047223 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.047030    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.047342    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.047660    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.047788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.047699    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.049554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.049533    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.049712 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.049693    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.102936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.102872    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.103054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.102943    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl45t\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-kube-api-access-dl45t\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.127186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.127155    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:03:58.180146 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.179784    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:03:58.514766 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.514735    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:03:58.555350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.555321    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"]
Oct 31 19:03:58.556683 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:03:58.556658    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4aa41cd_e710_49da_bc0c_56ffb03609fd.slice/crio-5f50ad4c4da946a8eb2db3794ec92c0e44ff5ae49211041332c89adf8e2a9fbd WatchSource:0}: Error finding container 5f50ad4c4da946a8eb2db3794ec92c0e44ff5ae49211041332c89adf8e2a9fbd: Status 404 returned error can't find the container with id 5f50ad4c4da946a8eb2db3794ec92c0e44ff5ae49211041332c89adf8e2a9fbd
Oct 31 19:03:58.559318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.559289    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:58.559396 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.559348    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:58.559396 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.559376    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:58.587960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.587936    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" event={"ID":"c4aa41cd-e710-49da-bc0c-56ffb03609fd","Type":"ContainerStarted","Data":"5f50ad4c4da946a8eb2db3794ec92c0e44ff5ae49211041332c89adf8e2a9fbd"}
Oct 31 19:03:58.608369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.608338    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"]
Oct 31 19:03:58.608660 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:03:58.608638    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741f42cf_57bd_455b_8fc5_7d0da0bd4682.slice/crio-398443ea72ab0ab5c77d71970eb614c9f515ea193d6dcd9e12a010b471e99f3d WatchSource:0}: Error finding container 398443ea72ab0ab5c77d71970eb614c9f515ea193d6dcd9e12a010b471e99f3d: Status 404 returned error can't find the container with id 398443ea72ab0ab5c77d71970eb614c9f515ea193d6dcd9e12a010b471e99f3d
Oct 31 19:03:58.611402 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.611367    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:58.611535 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.611477    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:58.611535 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:58.611519    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:03:59.606305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:59.606261    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" event={"ID":"c4aa41cd-e710-49da-bc0c-56ffb03609fd","Type":"ContainerStarted","Data":"c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8"}
Oct 31 19:03:59.608251 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:59.608226    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" event={"ID":"741f42cf-57bd-455b-8fc5-7d0da0bd4682","Type":"ContainerStarted","Data":"476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba"}
Oct 31 19:03:59.608363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:59.608271    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" event={"ID":"741f42cf-57bd-455b-8fc5-7d0da0bd4682","Type":"ContainerStarted","Data":"398443ea72ab0ab5c77d71970eb614c9f515ea193d6dcd9e12a010b471e99f3d"}
Oct 31 19:03:59.731265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:59.731215    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" podStartSLOduration=2.731201495 podStartE2EDuration="2.731201495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:59.729450641 +0000 UTC m=+2542.960688931" watchObservedRunningTime="2026-10-31 19:03:59.731201495 +0000 UTC m=+2542.962439785"
Oct 31 19:03:59.846642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:03:59.846587    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" podStartSLOduration=2.846568367 podStartE2EDuration="2.846568367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:59.84526034 +0000 UTC m=+2543.076498630" watchObservedRunningTime="2026-10-31 19:03:59.846568367 +0000 UTC m=+2543.077806661"
Oct 31 19:04:00.127602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:00.127575    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:04:00.128945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:00.128917    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 19:04:00.129099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:00.128969    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 19:04:00.200484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:00.180778    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:04:00.200484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:00.182761    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:04:00.200484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:00.182811    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 19:04:00.609394 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:00.607927    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:04:01.127888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:01.127845    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 19:04:01.128106 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:01.127922    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 19:04:01.183485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:01.183367    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:04:01.183485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:01.183426    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 19:04:02.128392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:02.128020    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 19:04:02.128392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:02.128081    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 19:04:02.180957 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:02.180920    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:04:02.181288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:02.181258    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 19:04:03.127865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:03.127837    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 19:04:03.128048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:03.127881    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 19:04:03.182486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:03.181878    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:04:03.182486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:03.181936    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 19:04:04.127500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:04.127435    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 19:04:04.127696 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:04.127533    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 19:04:04.187434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:04.187395    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:04:04.188056 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:04.188023    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 19:04:05.128070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:05.128036    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 19:04:05.128271 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:05.128094    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 19:04:05.181002 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:05.180964    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:04:05.181167 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:05.181034    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 19:04:06.139759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:06.139724    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:04:06.140326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:06.140163    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:04:06.142080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:06.142057    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:04:06.186871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:06.186843    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:04:06.187225 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:06.187197    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:04:06.188209 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:06.188180    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:04:16.604607 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:04:16.604569    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:04:18.788552 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:18.788508    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"]
Oct 31 19:04:18.794002 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:18.793978    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:18.846641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:18.846610    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a2c587-serving-cert\""
Oct 31 19:04:18.903217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:18.900787    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"]
Oct 31 19:04:18.925866 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:18.925838    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d366cb8-7755-423c-b137-92c62c614b43-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:18.926127 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:18.926109    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkmbn\" (UniqueName: \"kubernetes.io/projected/0d366cb8-7755-423c-b137-92c62c614b43-kube-api-access-mkmbn\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:18.926365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:18.926352    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:18.926503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:18.926492    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-metrics-certs\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.027759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.027729    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d366cb8-7755-423c-b137-92c62c614b43-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.027938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.027772    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmbn\" (UniqueName: \"kubernetes.io/projected/0d366cb8-7755-423c-b137-92c62c614b43-kube-api-access-mkmbn\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.027938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.027897    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.027938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.027921    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-metrics-certs\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.028872 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.028841    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d366cb8-7755-423c-b137-92c62c614b43-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.030692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.030664    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-metrics-certs\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.030959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.030943    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.090264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.090163    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmbn\" (UniqueName: \"kubernetes.io/projected/0d366cb8-7755-423c-b137-92c62c614b43-kube-api-access-mkmbn\") pod \"maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.109425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.109400    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:19.548095 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.548054    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"]
Oct 31 19:04:19.554783 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:04:19.554742    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d366cb8_7755_423c_b137_92c62c614b43.slice/crio-8ed64da670396789379cf95fc30575bbd617350302f7e818b7af3bc9a8ba89ef WatchSource:0}: Error finding container 8ed64da670396789379cf95fc30575bbd617350302f7e818b7af3bc9a8ba89ef: Status 404 returned error can't find the container with id 8ed64da670396789379cf95fc30575bbd617350302f7e818b7af3bc9a8ba89ef
Oct 31 19:04:19.768491 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:19.764828    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp" event={"ID":"0d366cb8-7755-423c-b137-92c62c614b43","Type":"ContainerStarted","Data":"8ed64da670396789379cf95fc30575bbd617350302f7e818b7af3bc9a8ba89ef"}
Oct 31 19:04:20.775828 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:20.775788    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp" event={"ID":"0d366cb8-7755-423c-b137-92c62c614b43","Type":"ContainerStarted","Data":"c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6"}
Oct 31 19:04:20.776298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:20.775870    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:20.892957 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:20.892903    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp" podStartSLOduration=2.052337643 podStartE2EDuration="2.892885022s" totalImagesPullingTime="840.547379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:20.891007343 +0000 UTC m=+2564.122245633" watchObservedRunningTime="2026-10-31 19:04:20.892885022 +0000 UTC m=+2564.124123312"
Oct 31 19:04:23.622041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.622002    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"]
Oct 31 19:04:23.626295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.626278    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.678287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.678261    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-88085e-openshift-default-dockercfg-96c8d\""
Oct 31 19:04:23.731693 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.731654    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"]
Oct 31 19:04:23.793658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.793628    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.793658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.793661    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zgx\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-kube-api-access-g9zgx\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.793858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.793684    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.793858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.793759    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.793932 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.793881    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.793970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.793928    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.793970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.793956    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.794122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.794094    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.794177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.794162    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.895539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895500    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.895539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895549    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.895794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895615    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.895794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895641    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.895794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895732    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.895794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895757    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zgx\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-kube-api-access-g9zgx\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.895794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895786    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.896078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895823    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.896078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.895897    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.896078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.896022    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.896311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.896290    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.896639 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.896611    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.896797 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.896746    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.896987 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.896958    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.898286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.898260    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.901539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.901518    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.953596 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.953560    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:23.954011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:23.953983    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zgx\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-kube-api-access-g9zgx\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:24.242620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.242591    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:24.710155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.710131    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"]
Oct 31 19:04:24.712345 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:04:24.712318    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e01880f_5ba4_4e71_bcf5_7ef30d06c4d5.slice/crio-dbfc4a5207d6ccba9cca81f746a784f7d08b0f6136c3a73d15c527a99647dd02 WatchSource:0}: Error finding container dbfc4a5207d6ccba9cca81f746a784f7d08b0f6136c3a73d15c527a99647dd02: Status 404 returned error can't find the container with id dbfc4a5207d6ccba9cca81f746a784f7d08b0f6136c3a73d15c527a99647dd02
Oct 31 19:04:24.714832 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.714796    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:24.714936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.714875    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:24.714936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.714917    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:24.805025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.804978    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" event={"ID":"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5","Type":"ContainerStarted","Data":"60ef5af612eb242c64ec8c9661c1af04b662f6d9bcec9ef0fe9d2c3486fb432c"}
Oct 31 19:04:24.805198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.805064    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" event={"ID":"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5","Type":"ContainerStarted","Data":"dbfc4a5207d6ccba9cca81f746a784f7d08b0f6136c3a73d15c527a99647dd02"}
Oct 31 19:04:24.807003 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.806968    5551 generic.go:356] "Generic (PLEG): container finished" podID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerID="03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490" exitCode=0
Oct 31 19:04:24.807126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.807033    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" event={"ID":"faa5d1dd-d82d-4484-acfb-c936f2088799","Type":"ContainerDied","Data":"03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490"}
Oct 31 19:04:24.807411 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.807398    5551 scope.go:122] "RemoveContainer" containerID="03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490"
Oct 31 19:04:24.808717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.808693    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:24.808782 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.808755    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:24.808823 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.808784    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:24.933073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:24.933020    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podStartSLOduration=1.933001549 podStartE2EDuration="1.933001549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:24.931653364 +0000 UTC m=+2568.162891656" watchObservedRunningTime="2026-10-31 19:04:24.933001549 +0000 UTC m=+2568.164239840"
Oct 31 19:04:25.050205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.050124    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"]
Oct 31 19:04:25.053959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.053935    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.117241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.114793    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-d8f34e-openshift-default-dockercfg-dmchr\""
Oct 31 19:04:25.120786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.120747    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"]
Oct 31 19:04:25.239038 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.238912    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-socket\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.239394 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.239367    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-data\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.239882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.239859    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c2c425-0d2f-4073-a253-808f42309478-istio-podinfo\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.240057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.240022    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-envoy\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.240262 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.240240    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62tpg\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-kube-api-access-62tpg\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.240400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.240382    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-istio-token\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.240720 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.240634    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c2c425-0d2f-4073-a253-808f42309478-istiod-ca-cert\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.240964 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.240942    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-credential-socket\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.241189 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.241171    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-certs\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.248241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.248214    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:25.262308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.253900    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:04:25.262308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.253957    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:04:25.342783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.342691    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c2c425-0d2f-4073-a253-808f42309478-istiod-ca-cert\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.342783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.342761    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-credential-socket\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.342788    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-certs\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.342820    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-socket\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.342869    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-data\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.342900    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c2c425-0d2f-4073-a253-808f42309478-istio-podinfo\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.342922    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-envoy\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.342974    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62tpg\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-kube-api-access-62tpg\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.343010    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-istio-token\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343424 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.343264    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-certs\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343424 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.343342    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-credential-socket\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.343531    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c2c425-0d2f-4073-a253-808f42309478-istiod-ca-cert\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.343604    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-data\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.343719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.343661    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-socket\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.346045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.346012    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-envoy\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.346231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.346209    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c2c425-0d2f-4073-a253-808f42309478-istio-podinfo\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.400674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.400575    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-istio-token\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.401611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.401555    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62tpg\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-kube-api-access-62tpg\") pod \"e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") " pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.409324 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.409300    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:25.824834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.824749    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" event={"ID":"faa5d1dd-d82d-4484-acfb-c936f2088799","Type":"ContainerStarted","Data":"28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b"}
Oct 31 19:04:25.898957 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.898926    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"]
Oct 31 19:04:25.912028 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:04:25.911958    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c2c425_0d2f_4073_a253_808f42309478.slice/crio-23bf3956859c0b8afb4f3b430b2f626498ebb9b8e27eeacfb23ea2b80adc2f28 WatchSource:0}: Error finding container 23bf3956859c0b8afb4f3b430b2f626498ebb9b8e27eeacfb23ea2b80adc2f28: Status 404 returned error can't find the container with id 23bf3956859c0b8afb4f3b430b2f626498ebb9b8e27eeacfb23ea2b80adc2f28
Oct 31 19:04:25.915692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.915659    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:25.915806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.915741    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:25.915806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:25.915783    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:26.243359 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:26.243326    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:04:26.243531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:26.243374    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:04:26.505873 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:26.505836    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:04:26.508471 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:26.508415    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:04:26.508723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:26.508682    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 19:04:26.802248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:26.801645    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:04:26.833690 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:26.833650    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" event={"ID":"77c2c425-0d2f-4073-a253-808f42309478","Type":"ContainerStarted","Data":"7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766"}
Oct 31 19:04:26.834133 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:26.833716    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" event={"ID":"77c2c425-0d2f-4073-a253-808f42309478","Type":"ContainerStarted","Data":"23bf3956859c0b8afb4f3b430b2f626498ebb9b8e27eeacfb23ea2b80adc2f28"}
Oct 31 19:04:27.243327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.243285    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:04:27.243593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.243341    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:04:27.409856 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.409807    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:27.416122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.416096    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:27.503535 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.502915    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:04:27.503535 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.502981    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 19:04:27.538665 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.538606    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" podStartSLOduration=2.53858773 podStartE2EDuration="2.53858773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:27.102807685 +0000 UTC m=+2570.334045977" watchObservedRunningTime="2026-10-31 19:04:27.53858773 +0000 UTC m=+2570.769826020"
Oct 31 19:04:27.840168 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.840091    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:27.840805 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:27.840787    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:04:28.243782 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:28.243744    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:04:28.243975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:28.243799    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:04:28.506114 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:28.506002    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:04:28.506114 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:28.506098    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 19:04:29.243192 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:29.243151    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:04:29.243621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:29.243210    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:04:29.503542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:29.502831    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:04:29.503542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:29.502898    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 19:04:30.237996 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:30.237952    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:04:30.243861 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:30.243833    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:04:30.244232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:30.243890    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:04:30.502833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:30.502745    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:04:30.502833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:30.502796    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 31 19:04:31.243384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:31.243342    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:04:31.243595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:31.243421    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:04:31.507027 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:31.506948    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:04:31.508124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:31.508103    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:04:32.364177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:32.364139    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:32.365152 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:32.365119    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:32.366281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:32.366253    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:04:33.220023 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.219991    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"]
Oct 31 19:04:33.224055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.224038    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.276177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.276147    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bbd8c6-openshift-default-dockercfg-kcjxp\""
Oct 31 19:04:33.326703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.326668    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"]
Oct 31 19:04:33.354214 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354185    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istiod-ca-cert\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.354408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354240    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-certs\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.354408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354353    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-socket\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.354408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354371    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-token\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.354408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354391    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-credential-socket\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.354697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354420    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-data\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.354697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354518    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-envoy\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.354697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354543    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-podinfo\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.354697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.354572    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-kube-api-access-g4qm2\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.455726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.455696    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-envoy\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.455933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.455735    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-podinfo\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.455933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.455764    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-kube-api-access-g4qm2\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.455933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.455822    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istiod-ca-cert\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.455933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.455879    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-certs\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.456170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.455943    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-socket\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.456170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.455969    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-token\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.456170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.455997    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-credential-socket\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.456170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.456041    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-data\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.456380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.456280    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-certs\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.456714 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.456659    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-credential-socket\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.456912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.456890    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istiod-ca-cert\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.456992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.456905    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-socket\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.457044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.457028    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-data\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.458391 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.458371    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-envoy\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.458706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.458689    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-podinfo\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.529256 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.529140    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-token\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.534563 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.534531    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-kube-api-access-g4qm2\") pod \"e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") " pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.549530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.547093    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:33.994736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:33.994705    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"]
Oct 31 19:04:34.033693 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:04:34.033664    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ff86d6_0892_42f9_b0b9_d3319a207fcb.slice/crio-62605799e230e3b9ce06b6415807b9f5c73fa4465da176e9236c682c1c76bd3f WatchSource:0}: Error finding container 62605799e230e3b9ce06b6415807b9f5c73fa4465da176e9236c682c1c76bd3f: Status 404 returned error can't find the container with id 62605799e230e3b9ce06b6415807b9f5c73fa4465da176e9236c682c1c76bd3f
Oct 31 19:04:34.036400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:34.036365    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:34.036527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:34.036452    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:34.036527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:34.036516    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:34.923007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:34.922968    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" event={"ID":"c3ff86d6-0892-42f9-b0b9-d3319a207fcb","Type":"ContainerStarted","Data":"94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315"}
Oct 31 19:04:34.923409 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:34.923030    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" event={"ID":"c3ff86d6-0892-42f9-b0b9-d3319a207fcb","Type":"ContainerStarted","Data":"62605799e230e3b9ce06b6415807b9f5c73fa4465da176e9236c682c1c76bd3f"}
Oct 31 19:04:35.139337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:35.139288    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podStartSLOduration=2.139274424 podStartE2EDuration="2.139274424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:35.088060948 +0000 UTC m=+2578.319299239" watchObservedRunningTime="2026-10-31 19:04:35.139274424 +0000 UTC m=+2578.370512714"
Oct 31 19:04:35.548285 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:35.548253    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:35.555317 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:35.555294    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:35.931075 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:35.931042    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:35.931927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:35.931908    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:04:39.487955 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:39.487917    5551 scope.go:122] "RemoveContainer" containerID="f8e3a49d6ad6ec06fbc48c68d38570c87b5cd0f5f3904449673654ef4131fa7e"
Oct 31 19:04:46.607952 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:04:46.607906    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:04:46.632870 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.632836    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"]
Oct 31 19:04:46.636992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.636972    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.688883 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.688849    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-dbceb9-openshift-default-dockercfg-6cqmr\""
Oct 31 19:04:46.751056 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.751018    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"]
Oct 31 19:04:46.800648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.800603    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e4dd78-4759-4bdb-9523-33ad3808763f-istiod-ca-cert\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.800999 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.800953    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-credential-socket\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.803743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.803715    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-token\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.803948 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.803932    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-data\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.804253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.804236    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-podinfo\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.804400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.804381    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-socket\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.804581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.804559    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-certs\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.804710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.804694    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczdj\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-kube-api-access-bczdj\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.804788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.804726    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-envoy\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.905929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.905890    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-podinfo\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.905929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.905942    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-socket\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.905970    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-certs\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906228    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczdj\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-kube-api-access-bczdj\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906257    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-envoy\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906417 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906363    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e4dd78-4759-4bdb-9523-33ad3808763f-istiod-ca-cert\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906417 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906393    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-credential-socket\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906439    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-token\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906502    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-data\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906545    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-socket\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906774 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906754    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-data\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.906973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.906955    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-credential-socket\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.907209 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.907176    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e4dd78-4759-4bdb-9523-33ad3808763f-istiod-ca-cert\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.907592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.907569    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-certs\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.909226 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.909193    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-envoy\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.910098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.910074    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-podinfo\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.964276 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.964243    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-token\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:46.964452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:46.964313    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczdj\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-kube-api-access-bczdj\") pod \"e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") " pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:47.251844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:47.251750    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:47.694166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:47.694136    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"]
Oct 31 19:04:47.696551 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:04:47.696518    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e4dd78_4759_4bdb_9523_33ad3808763f.slice/crio-5a996e6edc8085e43d8a81262eb0e5e81a8d176822252717b2bb05ad7e842855 WatchSource:0}: Error finding container 5a996e6edc8085e43d8a81262eb0e5e81a8d176822252717b2bb05ad7e842855: Status 404 returned error can't find the container with id 5a996e6edc8085e43d8a81262eb0e5e81a8d176822252717b2bb05ad7e842855
Oct 31 19:04:47.699269 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:47.699238    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:47.699331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:47.699305    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:47.699374 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:47.699341    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:48.040515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:48.040392    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" event={"ID":"f5e4dd78-4759-4bdb-9523-33ad3808763f","Type":"ContainerStarted","Data":"28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28"}
Oct 31 19:04:48.040515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:48.040485    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" event={"ID":"f5e4dd78-4759-4bdb-9523-33ad3808763f","Type":"ContainerStarted","Data":"5a996e6edc8085e43d8a81262eb0e5e81a8d176822252717b2bb05ad7e842855"}
Oct 31 19:04:48.158253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:48.158192    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podStartSLOduration=2.158173933 podStartE2EDuration="2.158173933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:48.155688617 +0000 UTC m=+2591.386926907" watchObservedRunningTime="2026-10-31 19:04:48.158173933 +0000 UTC m=+2591.389412224"
Oct 31 19:04:48.251964 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:48.251934    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:48.257899 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:48.257872    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:49.048752 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:49.048719    5551 generic.go:356] "Generic (PLEG): container finished" podID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerID="a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5" exitCode=0
Oct 31 19:04:49.049234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:49.048788    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" event={"ID":"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1","Type":"ContainerDied","Data":"a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5"}
Oct 31 19:04:49.049234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:49.049224    5551 scope.go:122] "RemoveContainer" containerID="a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5"
Oct 31 19:04:49.049375 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:49.049340    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:49.050374 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:49.050352    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:04:49.050751 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:49.050724    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:49.050844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:49.050809    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:49.050914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:49.050862    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:04:50.081263 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:50.081229    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" event={"ID":"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1","Type":"ContainerStarted","Data":"0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f"}
Oct 31 19:04:50.680643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:50.680609    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:04:50.681867 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:50.681842    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 19:04:50.682012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:50.681888    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 19:04:51.694487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:51.686946    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 19:04:51.694487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:51.687016    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 19:04:52.687490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:52.682814    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 19:04:52.687490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:52.682877    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 19:04:53.679732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:53.679693    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 19:04:53.680226 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:53.679758    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 19:04:54.403371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:54.403336    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:04:54.680845 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:54.680739    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 19:04:54.681499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:54.681451    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 19:04:55.683553 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:55.683522    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:04:55.684722 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:04:55.684701    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:05:08.241312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.241273    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"]
Oct 31 19:05:08.241819 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.241590    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8" podUID="b6360aaf-34aa-46f5-af0a-5191af0e5479" containerName="maas-api" containerID="cri-o://ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e" gracePeriod=30
Oct 31 19:05:08.589059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.589035    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:05:08.738842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.738800    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls\") pod \"b6360aaf-34aa-46f5-af0a-5191af0e5479\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") "
Oct 31 19:05:08.739048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.738910    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6360aaf-34aa-46f5-af0a-5191af0e5479-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6360aaf-34aa-46f5-af0a-5191af0e5479-openshift-service-ca\") pod \"b6360aaf-34aa-46f5-af0a-5191af0e5479\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") "
Oct 31 19:05:08.739136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.739092    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-metrics-certs\") pod \"b6360aaf-34aa-46f5-af0a-5191af0e5479\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") "
Oct 31 19:05:08.739195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.739153    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6360aaf-34aa-46f5-af0a-5191af0e5479-kube-api-access-bn945\" (UniqueName: \"kubernetes.io/projected/b6360aaf-34aa-46f5-af0a-5191af0e5479-kube-api-access-bn945\") pod \"b6360aaf-34aa-46f5-af0a-5191af0e5479\" (UID: \"b6360aaf-34aa-46f5-af0a-5191af0e5479\") "
Oct 31 19:05:08.740429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.740369    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6360aaf-34aa-46f5-af0a-5191af0e5479-openshift-service-ca" pod "b6360aaf-34aa-46f5-af0a-5191af0e5479" (UID: "b6360aaf-34aa-46f5-af0a-5191af0e5479"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:05:08.743696 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.743673    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls" pod "b6360aaf-34aa-46f5-af0a-5191af0e5479" (UID: "b6360aaf-34aa-46f5-af0a-5191af0e5479"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:05:08.748031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.748006    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-metrics-certs" pod "b6360aaf-34aa-46f5-af0a-5191af0e5479" (UID: "b6360aaf-34aa-46f5-af0a-5191af0e5479"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:05:08.754725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.754673    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6360aaf-34aa-46f5-af0a-5191af0e5479-kube-api-access-bn945" pod "b6360aaf-34aa-46f5-af0a-5191af0e5479" (UID: "b6360aaf-34aa-46f5-af0a-5191af0e5479"). InnerVolumeSpecName "kube-api-access-bn945". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:05:08.841757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.841635    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:05:08.841757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.841671    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn945\" (UniqueName: \"kubernetes.io/projected/b6360aaf-34aa-46f5-af0a-5191af0e5479-kube-api-access-bn945\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:05:08.841757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.841687    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6360aaf-34aa-46f5-af0a-5191af0e5479-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:05:08.841757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.841699    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6360aaf-34aa-46f5-af0a-5191af0e5479-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:05:08.969090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.969061    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"]
Oct 31 19:05:08.969753 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.969731    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b6360aaf-34aa-46f5-af0a-5191af0e5479" containerName="maas-api"
Oct 31 19:05:08.969753 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.969753    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6360aaf-34aa-46f5-af0a-5191af0e5479" containerName="maas-api"
Oct 31 19:05:08.969938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.969912    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6360aaf-34aa-46f5-af0a-5191af0e5479" containerName="maas-api"
Oct 31 19:05:08.973386 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:08.973365    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.031619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.031592    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"]
Oct 31 19:05:09.043647 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.043623    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.043801 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.043681    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.043801 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.043736    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljx64\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-kube-api-access-ljx64\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.043801 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.043772    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.043995 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.043965    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.044274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.044253    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.044568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.044544    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.044739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.044702    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.044812 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.044787    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146103    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146169    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146215    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146253    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146284    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljx64\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-kube-api-access-ljx64\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146328    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146367    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146442    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146633    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146682    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146809    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.146998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.146973    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.147201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.147176    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.147700 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.147679    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.148654 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.148630    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.148879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.148859    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.222187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.222112    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljx64\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-kube-api-access-ljx64\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.222813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.222762    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.280400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.276871    5551 generic.go:356] "Generic (PLEG): container finished" podID="b6360aaf-34aa-46f5-af0a-5191af0e5479" containerID="ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e" exitCode=0
Oct 31 19:05:09.280400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.276924    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"
Oct 31 19:05:09.280400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.277059    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8" event={"ID":"b6360aaf-34aa-46f5-af0a-5191af0e5479","Type":"ContainerDied","Data":"ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e"}
Oct 31 19:05:09.280400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.277099    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8" event={"ID":"b6360aaf-34aa-46f5-af0a-5191af0e5479","Type":"ContainerDied","Data":"4d93150fdf57f80f00cced0a9aae8b2273926674741b9b30f6a18c45661b9c29"}
Oct 31 19:05:09.280400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.277123    5551 scope.go:122] "RemoveContainer" containerID="ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e"
Oct 31 19:05:09.288831 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.288723    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:09.293912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.293888    5551 scope.go:122] "RemoveContainer" containerID="ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e"
Oct 31 19:05:09.294251 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:05:09.294228    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e\": container with ID starting with ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e not found: ID does not exist" containerID="ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e"
Oct 31 19:05:09.294343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.294261    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e"} err="failed to get container status \"ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e\": rpc error: code = NotFound desc = could not find container \"ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e\": container with ID starting with ce423341cadd59a004bd7b65e59e78bd4c2ece8b43e656de154a879febed945e not found: ID does not exist"
Oct 31 19:05:09.441942 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.441848    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"]
Oct 31 19:05:09.444784 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.444762    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-79e7b6-bfc74dd9c-mpgt8"]
Oct 31 19:05:09.717093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.717064    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"]
Oct 31 19:05:09.718655 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:05:09.718620    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bff25e_b2ff_42a7_bcff_c559781d1d9d.slice/crio-3b6e88cbbc9483a6cb19a7b2a24718e57d42d95d607ef6d39c0ef19e735f2a00 WatchSource:0}: Error finding container 3b6e88cbbc9483a6cb19a7b2a24718e57d42d95d607ef6d39c0ef19e735f2a00: Status 404 returned error can't find the container with id 3b6e88cbbc9483a6cb19a7b2a24718e57d42d95d607ef6d39c0ef19e735f2a00
Oct 31 19:05:09.721613 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.721579    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:09.721694 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.721654    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:09.721694 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.721686    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:09.841949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.841914    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"]
Oct 31 19:05:09.845917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.845890    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:09.900960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.900933    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-d8f34e-serving-cert\""
Oct 31 19:05:09.903371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.903336    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"]
Oct 31 19:05:09.958975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.958943    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:09.959171 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.958992    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2j9t\" (UniqueName: \"kubernetes.io/projected/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-kube-api-access-h2j9t\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:09.959171 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.959125    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:09.959291 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:09.959184    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.060937 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.060840    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.060937 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.060902    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2j9t\" (UniqueName: \"kubernetes.io/projected/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-kube-api-access-h2j9t\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.061175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.060949    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.061175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.061011    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.061690 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:05:10.061404    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-serving-cert: secret "maas-api-e2e-shared-a-w0-d8f34e-serving-cert" not found
Oct 31 19:05:10.061690 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:05:10.061488    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls podName:c962f3f6-c8e1-4e5b-8e77-512a5e7879f8 nodeName:}" failed. No retries permitted until 2026-10-31 19:05:10.561451029 +0000 UTC m=+2613.792689312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls") pod "maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv" (UID: "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8") : secret "maas-api-e2e-shared-a-w0-d8f34e-serving-cert" not found
Oct 31 19:05:10.062399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.062352    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.066019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.065993    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.123705 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.122919    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2j9t\" (UniqueName: \"kubernetes.io/projected/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-kube-api-access-h2j9t\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.290218 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.290177    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" event={"ID":"d0bff25e-b2ff-42a7-bcff-c559781d1d9d","Type":"ContainerStarted","Data":"063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96"}
Oct 31 19:05:10.290735 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.290273    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" event={"ID":"d0bff25e-b2ff-42a7-bcff-c559781d1d9d","Type":"ContainerStarted","Data":"3b6e88cbbc9483a6cb19a7b2a24718e57d42d95d607ef6d39c0ef19e735f2a00"}
Oct 31 19:05:10.410347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.410292    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podStartSLOduration=2.410276563 podStartE2EDuration="2.410276563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:10.407420255 +0000 UTC m=+2613.638658545" watchObservedRunningTime="2026-10-31 19:05:10.410276563 +0000 UTC m=+2613.641514915"
Oct 31 19:05:10.578570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.578529    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.581758 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.581727    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.770531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.769066    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:10.912375 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:10.912331    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6360aaf-34aa-46f5-af0a-5191af0e5479" path="/var/lib/kubelet/pods/b6360aaf-34aa-46f5-af0a-5191af0e5479/volumes"
Oct 31 19:05:11.161025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:11.160564    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"]
Oct 31 19:05:11.179099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:11.179065    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:05:11.289337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:11.289300    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:11.291264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:11.290952    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:05:11.291264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:11.291006    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:05:11.298747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:11.298715    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv" event={"ID":"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8","Type":"ContainerStarted","Data":"9f7ea9c3166eef8591310db094c438c48b6ac612028bf4c1519c00b30e46945c"}
Oct 31 19:05:12.289904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:12.289865    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:05:12.290084 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:12.289923    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:05:12.307674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:12.307636    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv" event={"ID":"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8","Type":"ContainerStarted","Data":"cb0cab60c64867baf3cf8dfa3be97537849d9c8c567d695fed0fb230ad692323"}
Oct 31 19:05:12.308074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:12.307710    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:12.444527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:12.444442    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv" podStartSLOduration=2.526183608 podStartE2EDuration="3.444424162s" totalImagesPullingTime="918.240554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:12.438719581 +0000 UTC m=+2615.669957879" watchObservedRunningTime="2026-10-31 19:05:12.444424162 +0000 UTC m=+2615.675662454"
Oct 31 19:05:13.289382 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:13.289329    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:05:13.289741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:13.289403    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:05:14.290080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:14.290041    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:05:14.290568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:14.290114    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:05:15.289976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:15.289934    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:05:15.290221 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:15.290016    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:05:16.289333 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:16.289296    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:05:16.289664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:16.289633    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:05:16.610621 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:05:16.610528    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:05:17.294161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:17.294123    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:17.294616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:17.294586    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:17.295494 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:17.295439    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:05:18.320721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:18.320685    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:05:25.250697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.250667    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"]
Oct 31 19:05:25.256160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.256140    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.329910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.329866    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.330161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.330137    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.330239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.330195    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777b590-d839-4db9-a4c4-8cb87403f3b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.330312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.330297    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvqwr\" (UniqueName: \"kubernetes.io/projected/a777b590-d839-4db9-a4c4-8cb87403f3b7-kube-api-access-wvqwr\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.358197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.358165    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-bbd8c6-serving-cert\""
Oct 31 19:05:25.361602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.361574    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"]
Oct 31 19:05:25.431818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.431786    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.432013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.431841    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777b590-d839-4db9-a4c4-8cb87403f3b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.432013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.431884    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvqwr\" (UniqueName: \"kubernetes.io/projected/a777b590-d839-4db9-a4c4-8cb87403f3b7-kube-api-access-wvqwr\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.432013 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:05:25.431952    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-serving-cert: secret "maas-api-e2e-shared-a-w3-bbd8c6-serving-cert" not found
Oct 31 19:05:25.432013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.431984    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.432250 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:05:25.432016    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls podName:a777b590-d839-4db9-a4c4-8cb87403f3b7 nodeName:}" failed. No retries permitted until 2026-10-31 19:05:25.931997816 +0000 UTC m=+2629.163236102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls") pod "maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp" (UID: "a777b590-d839-4db9-a4c4-8cb87403f3b7") : secret "maas-api-e2e-shared-a-w3-bbd8c6-serving-cert" not found
Oct 31 19:05:25.432645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.432624    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777b590-d839-4db9-a4c4-8cb87403f3b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.434531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.434509    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.491178 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.491140    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvqwr\" (UniqueName: \"kubernetes.io/projected/a777b590-d839-4db9-a4c4-8cb87403f3b7-kube-api-access-wvqwr\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.938844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.938812    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:25.941597 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:25.941573    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:26.172169 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:26.172124    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:26.553720 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:26.553683    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"]
Oct 31 19:05:26.568012 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:05:26.567981    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda777b590_d839_4db9_a4c4_8cb87403f3b7.slice/crio-cebb1b2e9aaffee97a4104bc5c42280eacd6721818e0eba9bb2ff1c876915624 WatchSource:0}: Error finding container cebb1b2e9aaffee97a4104bc5c42280eacd6721818e0eba9bb2ff1c876915624: Status 404 returned error can't find the container with id cebb1b2e9aaffee97a4104bc5c42280eacd6721818e0eba9bb2ff1c876915624
Oct 31 19:05:27.413931 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:27.413891    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp" event={"ID":"a777b590-d839-4db9-a4c4-8cb87403f3b7","Type":"ContainerStarted","Data":"cebb1b2e9aaffee97a4104bc5c42280eacd6721818e0eba9bb2ff1c876915624"}
Oct 31 19:05:28.437973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:28.437931    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp" event={"ID":"a777b590-d839-4db9-a4c4-8cb87403f3b7","Type":"ContainerStarted","Data":"6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b"}
Oct 31 19:05:28.439671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:28.439640    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:28.559238 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:28.559185    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp" podStartSLOduration=2.749764684 podStartE2EDuration="3.559168865s" totalImagesPullingTime="809.404181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:28.556105546 +0000 UTC m=+2631.787343839" watchObservedRunningTime="2026-10-31 19:05:28.559168865 +0000 UTC m=+2631.790407157"
Oct 31 19:05:35.456858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:35.456824    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:05:46.632794 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:05:46.632752    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:05:51.971829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:51.971589    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:05:51.971829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:51.971665    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:05:55.682654 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.682619    5551 generic.go:356] "Generic (PLEG): container finished" podID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerID="94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315" exitCode=0
Oct 31 19:05:55.683202 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.682703    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" event={"ID":"c3ff86d6-0892-42f9-b0b9-d3319a207fcb","Type":"ContainerDied","Data":"94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315"}
Oct 31 19:05:55.683292 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.683198    5551 scope.go:122] "RemoveContainer" containerID="94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315"
Oct 31 19:05:55.684837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.684816    5551 generic.go:356] "Generic (PLEG): container finished" podID="77c2c425-0d2f-4073-a253-808f42309478" containerID="7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766" exitCode=0
Oct 31 19:05:55.684988 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.684870    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" event={"ID":"77c2c425-0d2f-4073-a253-808f42309478","Type":"ContainerDied","Data":"7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766"}
Oct 31 19:05:55.685154 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.685119    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:55.685229 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.685217    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:55.685283 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.685269    5551 scope.go:122] "RemoveContainer" containerID="7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766"
Oct 31 19:05:55.685340 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.685286    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:55.687046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.687010    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:55.687174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.687092    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:55.687174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:55.687141    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:05:56.698860 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:56.698820    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" event={"ID":"77c2c425-0d2f-4073-a253-808f42309478","Type":"ContainerStarted","Data":"f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118"}
Oct 31 19:05:56.703945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:56.703886    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" event={"ID":"c3ff86d6-0892-42f9-b0b9-d3319a207fcb","Type":"ContainerStarted","Data":"d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b"}
Oct 31 19:05:57.413483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:57.412974    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:05:57.413483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:57.413127    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:05:57.413483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:57.413171    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:05:57.547939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:57.547909    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:05:57.548126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:57.548104    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:05:57.548173 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:57.548151    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:05:57.840812 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:57.840729    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:05:58.409993 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:58.409956    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:05:58.410339 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:58.410289    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:05:58.548583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:58.548544    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:05:58.548794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:58.548616    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:05:59.410483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:59.409895    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:05:59.410483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:59.409961    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:05:59.548173 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:59.548129    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:05:59.548354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:05:59.548194    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:06:00.409998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:00.409955    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:06:00.410308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:00.410273    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:06:00.548508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:00.548474    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:06:00.548887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:00.548527    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:06:01.411501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:01.410929    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:06:01.411501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:01.411009    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:06:01.547971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:01.547942    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:06:01.548135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:01.547984    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:06:02.410533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:02.410479    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:06:02.411114 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:02.410556    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:06:02.547740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:02.547703    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:06:02.547945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:02.547780    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:06:03.415109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:03.415076    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:06:03.416227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:03.416208    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:06:03.559501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:03.558123    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:06:03.559501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:03.559305    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:06:03.571971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:03.571851    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:06:12.841434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.841396    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"]
Oct 31 19:06:12.844397 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.844374    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.882249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.879675    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6nf\" (UniqueName: \"kubernetes.io/projected/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-kube-api-access-sc6nf\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.882249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.879828    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.882249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.879887    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.882249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.880054    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.892338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.890669    5551 generic.go:356] "Generic (PLEG): container finished" podID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerID="28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28" exitCode=0
Oct 31 19:06:12.892338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.890835    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" event={"ID":"f5e4dd78-4759-4bdb-9523-33ad3808763f","Type":"ContainerDied","Data":"28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28"}
Oct 31 19:06:12.892338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.891331    5551 scope.go:122] "RemoveContainer" containerID="28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28"
Oct 31 19:06:12.894829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.894773    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:12.895064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.895041    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:12.896652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.896625    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:12.929093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.927510    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"]
Oct 31 19:06:12.962020 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.949837    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7006d3-serving-cert\""
Oct 31 19:06:12.983778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.983291    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.983778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.983359    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6nf\" (UniqueName: \"kubernetes.io/projected/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-kube-api-access-sc6nf\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.987211 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.984083    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.987211 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.984157    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.987211 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:12.985275    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:12.988586 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:06:12.987779    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-serving-cert: secret "maas-api-e2e-praxis-b-7006d3-serving-cert" not found
Oct 31 19:06:12.988586 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:06:12.987858    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls podName:a97b5e09-e843-4853-b7f6-8c1ac22d6f8b nodeName:}" failed. No retries permitted until 2026-10-31 19:06:13.487836955 +0000 UTC m=+2676.719075237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls") pod "maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h" (UID: "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b") : secret "maas-api-e2e-praxis-b-7006d3-serving-cert" not found
Oct 31 19:06:13.017778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:13.017742    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:13.045201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:13.045174    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6nf\" (UniqueName: \"kubernetes.io/projected/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-kube-api-access-sc6nf\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:13.492175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:13.492133    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:13.495012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:13.494982    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:13.536674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:13.536647    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:13.904929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:13.904802    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" event={"ID":"f5e4dd78-4759-4bdb-9523-33ad3808763f","Type":"ContainerStarted","Data":"4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa"}
Oct 31 19:06:13.929254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:13.929224    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"]
Oct 31 19:06:13.929821 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:06:13.929796    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97b5e09_e843_4853_b7f6_8c1ac22d6f8b.slice/crio-06e95589cf86c8864ddeaae8c1a46ac14bdffdc42c4a827fb22797465874bc14 WatchSource:0}: Error finding container 06e95589cf86c8864ddeaae8c1a46ac14bdffdc42c4a827fb22797465874bc14: Status 404 returned error can't find the container with id 06e95589cf86c8864ddeaae8c1a46ac14bdffdc42c4a827fb22797465874bc14
Oct 31 19:06:14.271541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:14.271316    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:06:14.271541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:14.271378    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:06:14.271541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:14.271513    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:06:14.916079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:14.916040    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h" event={"ID":"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b","Type":"ContainerStarted","Data":"06e95589cf86c8864ddeaae8c1a46ac14bdffdc42c4a827fb22797465874bc14"}
Oct 31 19:06:15.265534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:15.253764    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:06:15.265534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:15.253819    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:06:15.933286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:15.933248    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h" event={"ID":"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b","Type":"ContainerStarted","Data":"94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31"}
Oct 31 19:06:15.933808 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:15.933350    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:16.287077 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:16.271092    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:06:16.287077 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:16.286589    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:06:16.611838 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:06:16.611751    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:06:17.255801 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:17.255764    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:06:17.256385 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:17.256353    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:06:17.639659 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:17.639608    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h" podStartSLOduration=4.641594002 podStartE2EDuration="5.639591878s" totalImagesPullingTime="997.997876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:16.070395443 +0000 UTC m=+2679.301633746" watchObservedRunningTime="2026-10-31 19:06:17.639591878 +0000 UTC m=+2680.870830192"
Oct 31 19:06:17.639840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:17.639744    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"]
Oct 31 19:06:17.639939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:17.639922    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc" podUID="ca526226-de23-4bd9-b474-162919d2af8e" containerName="maas-api" containerID="cri-o://fb305087d2a2f638ccf83e2696dd9a8fc9166ae028546c454ae0f730d9726ecb" gracePeriod=30
Oct 31 19:06:18.064629 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.061804    5551 generic.go:356] "Generic (PLEG): container finished" podID="ca526226-de23-4bd9-b474-162919d2af8e" containerID="fb305087d2a2f638ccf83e2696dd9a8fc9166ae028546c454ae0f730d9726ecb" exitCode=0
Oct 31 19:06:18.064629 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.062283    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc" event={"ID":"ca526226-de23-4bd9-b474-162919d2af8e","Type":"ContainerDied","Data":"fb305087d2a2f638ccf83e2696dd9a8fc9166ae028546c454ae0f730d9726ecb"}
Oct 31 19:06:18.286483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.285020    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:06:18.286483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.285078    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:06:18.294934 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.294912    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:06:18.309993 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.309965    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls\") pod \"ca526226-de23-4bd9-b474-162919d2af8e\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") "
Oct 31 19:06:18.310183 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.310162    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca526226-de23-4bd9-b474-162919d2af8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca526226-de23-4bd9-b474-162919d2af8e-openshift-service-ca\") pod \"ca526226-de23-4bd9-b474-162919d2af8e\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") "
Oct 31 19:06:18.310319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.310308    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca526226-de23-4bd9-b474-162919d2af8e-kube-api-access-jpdfr\" (UniqueName: \"kubernetes.io/projected/ca526226-de23-4bd9-b474-162919d2af8e-kube-api-access-jpdfr\") pod \"ca526226-de23-4bd9-b474-162919d2af8e\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") "
Oct 31 19:06:18.310447 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.310437    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-metrics-certs\") pod \"ca526226-de23-4bd9-b474-162919d2af8e\" (UID: \"ca526226-de23-4bd9-b474-162919d2af8e\") "
Oct 31 19:06:18.310606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.310580    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca526226-de23-4bd9-b474-162919d2af8e-openshift-service-ca" pod "ca526226-de23-4bd9-b474-162919d2af8e" (UID: "ca526226-de23-4bd9-b474-162919d2af8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:06:18.311132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.311116    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca526226-de23-4bd9-b474-162919d2af8e-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:18.312177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.312152    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls" pod "ca526226-de23-4bd9-b474-162919d2af8e" (UID: "ca526226-de23-4bd9-b474-162919d2af8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:06:18.312570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.312548    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca526226-de23-4bd9-b474-162919d2af8e-kube-api-access-jpdfr" pod "ca526226-de23-4bd9-b474-162919d2af8e" (UID: "ca526226-de23-4bd9-b474-162919d2af8e"). InnerVolumeSpecName "kube-api-access-jpdfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:06:18.314103 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.314079    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-metrics-certs" pod "ca526226-de23-4bd9-b474-162919d2af8e" (UID: "ca526226-de23-4bd9-b474-162919d2af8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:06:18.412776 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.412740    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:18.412776 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.412772    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdfr\" (UniqueName: \"kubernetes.io/projected/ca526226-de23-4bd9-b474-162919d2af8e-kube-api-access-jpdfr\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:18.412776 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:18.412783    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca526226-de23-4bd9-b474-162919d2af8e-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:19.050600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:19.050567    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:06:19.088840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:19.088807    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"
Oct 31 19:06:19.089019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:19.088866    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc" event={"ID":"ca526226-de23-4bd9-b474-162919d2af8e","Type":"ContainerDied","Data":"eeb47bc96bc22c246d500e4a1f32534eb01b8d8932c6961a63d34e8cf2848b02"}
Oct 31 19:06:19.089019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:19.088951    5551 scope.go:122] "RemoveContainer" containerID="fb305087d2a2f638ccf83e2696dd9a8fc9166ae028546c454ae0f730d9726ecb"
Oct 31 19:06:19.265540 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:19.263783    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:06:19.265540 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:19.263837    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:06:19.281482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:19.280588    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"]
Oct 31 19:06:19.307005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:19.306971    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1ef01f-7554454d8f-5vmwc"]
Oct 31 19:06:20.256852 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:20.256822    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:06:20.258037 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:20.258006    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:06:20.910411 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:20.910381    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca526226-de23-4bd9-b474-162919d2af8e" path="/var/lib/kubelet/pods/ca526226-de23-4bd9-b474-162919d2af8e/volumes"
Oct 31 19:06:21.945392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:21.945358    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:06:26.415993 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.415957    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"]
Oct 31 19:06:26.416570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.416556    5551 cpu_manager.go:414] "containerMap: removing container" podUID="ca526226-de23-4bd9-b474-162919d2af8e" containerName="maas-api"
Oct 31 19:06:26.416626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.416571    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca526226-de23-4bd9-b474-162919d2af8e" containerName="maas-api"
Oct 31 19:06:26.416679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.416670    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca526226-de23-4bd9-b474-162919d2af8e" containerName="maas-api"
Oct 31 19:06:26.419618 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.419601    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.582845 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.582812    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d22e36-openshift-default-dockercfg-hfrrt\""
Oct 31 19:06:26.585937 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.583382    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-data\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.586111 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.586003    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-socket\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.586111 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.586040    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6h9\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-kube-api-access-mh6h9\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.586111 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.586100    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-certs\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.586293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.586162    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-credential-socket\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.586293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.586192    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-envoy\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.586293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.586267    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-token\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.586475 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.586349    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-podinfo\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.586550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.586509    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f2799-c8ee-418d-ba27-31d90cba3a40-istiod-ca-cert\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.592850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.592818    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"]
Oct 31 19:06:26.635399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.635371    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"]
Oct 31 19:06:26.635637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.635610    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy" containerID="cri-o://28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b" gracePeriod=30
Oct 31 19:06:26.687989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.687905    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-data\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688137 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.687990    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-socket\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688137 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688025    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6h9\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-kube-api-access-mh6h9\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688137 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688085    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-certs\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688232    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-credential-socket\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688286    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-envoy\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688376    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-data\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688395    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-token\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688410    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-socket\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688495    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-certs\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688531    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-podinfo\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688562    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f2799-c8ee-418d-ba27-31d90cba3a40-istiod-ca-cert\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.688623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.688593    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-credential-socket\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.689238 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.689159    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f2799-c8ee-418d-ba27-31d90cba3a40-istiod-ca-cert\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.690692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.690663    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-envoy\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.691048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.691028    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-podinfo\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.796324 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.796288    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6h9\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-kube-api-access-mh6h9\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:26.825479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:26.823036    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-token\") pod \"e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") " pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:27.031638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:27.031542    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:27.464713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:27.464684    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"]
Oct 31 19:06:27.465778 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:06:27.465751    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32f2799_c8ee_418d_ba27_31d90cba3a40.slice/crio-55897f8a4f653fd4ace1249a50e6deee3ad0ebc7e66d57b13695a4a64333651f WatchSource:0}: Error finding container 55897f8a4f653fd4ace1249a50e6deee3ad0ebc7e66d57b13695a4a64333651f: Status 404 returned error can't find the container with id 55897f8a4f653fd4ace1249a50e6deee3ad0ebc7e66d57b13695a4a64333651f
Oct 31 19:06:27.468248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:27.468215    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:27.468318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:27.468280    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:27.468318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:27.468308    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:28.205252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:28.205220    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" event={"ID":"d32f2799-c8ee-418d-ba27-31d90cba3a40","Type":"ContainerStarted","Data":"e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0"}
Oct 31 19:06:28.205498 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:28.205276    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" event={"ID":"d32f2799-c8ee-418d-ba27-31d90cba3a40","Type":"ContainerStarted","Data":"55897f8a4f653fd4ace1249a50e6deee3ad0ebc7e66d57b13695a4a64333651f"}
Oct 31 19:06:28.380247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:28.380194    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" podStartSLOduration=2.38017848 podStartE2EDuration="2.38017848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:28.368816158 +0000 UTC m=+2691.600054452" watchObservedRunningTime="2026-10-31 19:06:28.38017848 +0000 UTC m=+2691.611416769"
Oct 31 19:06:29.032347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:29.032306    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:29.039989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:29.039942    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:29.213728 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:29.213695    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:29.214581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:29.214557    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:06:31.510378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:31.510349    5551 patch_prober.go:29] interesting pod/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:06:31.510763 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:31.510413    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:06:31.920195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:31.920166    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:06:32.065287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065202    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-envoy\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.065287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065258    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-data\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.065547 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065292    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-token\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-token\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.065547 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065347    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-socket\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.065547 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065407    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-credential-socket\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.065547 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065469    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-kube-api-access-xw6mp\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-kube-api-access-xw6mp\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.065547 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065538    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-podinfo\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.065874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065555    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-data" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:06:32.065874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065628    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa5d1dd-d82d-4484-acfb-c936f2088799-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa5d1dd-d82d-4484-acfb-c936f2088799-istiod-ca-cert\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.065874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065720    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-credential-socket" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:06:32.065874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.065841    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-certs\") pod \"faa5d1dd-d82d-4484-acfb-c936f2088799\" (UID: \"faa5d1dd-d82d-4484-acfb-c936f2088799\") "
Oct 31 19:06:32.067564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.066602    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.067564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.066639    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.067564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.067014    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-certs" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:06:32.067818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.067651    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa5d1dd-d82d-4484-acfb-c936f2088799-istiod-ca-cert" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:06:32.067879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.067823    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-socket" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:06:32.077412 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.077383    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-kube-api-access-xw6mp" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "kube-api-access-xw6mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:06:32.078813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.078786    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-token" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:06:32.078902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.078869    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-envoy" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:06:32.079265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.079239    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-podinfo" pod "faa5d1dd-d82d-4484-acfb-c936f2088799" (UID: "faa5d1dd-d82d-4484-acfb-c936f2088799"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:06:32.168411 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.168373    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.168411 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.168411    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw6mp\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-kube-api-access-xw6mp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.168636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.168426    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.168636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.168439    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa5d1dd-d82d-4484-acfb-c936f2088799-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.168636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.168470    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.168636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.168485    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.168636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.168498    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa5d1dd-d82d-4484-acfb-c936f2088799-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:06:32.237192 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.237158    5551 generic.go:356] "Generic (PLEG): container finished" podID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerID="28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b" exitCode=0
Oct 31 19:06:32.237367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.237209    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"
Oct 31 19:06:32.237367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.237224    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" event={"ID":"faa5d1dd-d82d-4484-acfb-c936f2088799","Type":"ContainerDied","Data":"28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b"}
Oct 31 19:06:32.237367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.237276    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47" event={"ID":"faa5d1dd-d82d-4484-acfb-c936f2088799","Type":"ContainerDied","Data":"b286b588d701853449ed281194af19abed4c425487c4bfc7795536e48e1f7474"}
Oct 31 19:06:32.237367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.237307    5551 scope.go:122] "RemoveContainer" containerID="28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b"
Oct 31 19:06:32.248530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.248512    5551 scope.go:122] "RemoveContainer" containerID="03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490"
Oct 31 19:06:32.259945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.259923    5551 scope.go:122] "RemoveContainer" containerID="28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b"
Oct 31 19:06:32.260222 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:06:32.260203    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b\": container with ID starting with 28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b not found: ID does not exist" containerID="28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b"
Oct 31 19:06:32.260272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.260231    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b"} err="failed to get container status \"28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b\": rpc error: code = NotFound desc = could not find container \"28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b\": container with ID starting with 28880a472853c32e3e639c3ad290e3f6db0aabc4ec2c1c37a4b6cddba2c03e1b not found: ID does not exist"
Oct 31 19:06:32.260272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.260251    5551 scope.go:122] "RemoveContainer" containerID="03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490"
Oct 31 19:06:32.260512 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:06:32.260495    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490\": container with ID starting with 03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490 not found: ID does not exist" containerID="03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490"
Oct 31 19:06:32.260575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.260517    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490"} err="failed to get container status \"03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490\": rpc error: code = NotFound desc = could not find container \"03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490\": container with ID starting with 03dd03fd2b3370448bef74b7fa170bc175178e689e2edbbdced88df1d10a2490 not found: ID does not exist"
Oct 31 19:06:32.402405 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.402369    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"]
Oct 31 19:06:32.405704 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.405679    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1ef01f-openshift-default-897f7bfb9-9bs47"]
Oct 31 19:06:32.918908 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:32.918870    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" path="/var/lib/kubelet/pods/faa5d1dd-d82d-4484-acfb-c936f2088799/volumes"
Oct 31 19:06:40.238885 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:40.238855    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:06:40.240279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:40.240256    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:06:44.167451 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.167363    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"]
Oct 31 19:06:44.168299 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.168277    5551 cpu_manager.go:414] "containerMap: removing container" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy"
Oct 31 19:06:44.168347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.168303    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy"
Oct 31 19:06:44.168388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.168345    5551 cpu_manager.go:414] "containerMap: removing container" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy"
Oct 31 19:06:44.168388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.168354    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy"
Oct 31 19:06:44.168561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.168548    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy"
Oct 31 19:06:44.168612 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.168569    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa5d1dd-d82d-4484-acfb-c936f2088799" containerName="istio-proxy"
Oct 31 19:06:44.174396 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.174370    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.232580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.228513    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6cfd68-openshift-default-dockercfg-bfq85\""
Oct 31 19:06:44.250358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.250283    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7315b844-4c65-4097-9656-ed99e582ce4c-istio-podinfo\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.250757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.250733    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-kube-api-access-qdwfh\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.250970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.250951    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7315b844-4c65-4097-9656-ed99e582ce4c-istiod-ca-cert\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.251159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.251142    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-certs\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.251410 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.251388    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-istio-token\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.251618 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.251592    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.251801 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.251767    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-credential-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.252068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.252051    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-data\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.252198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.252184    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-envoy\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.330490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.328481    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"]
Oct 31 19:06:44.353383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.353297    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.353733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.353707    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-credential-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.353897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.353881    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-data\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.357541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.357254    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-credential-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.369487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.360802    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-envoy\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.369487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.360923    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7315b844-4c65-4097-9656-ed99e582ce4c-istio-podinfo\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.369487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.360957    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-kube-api-access-qdwfh\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.369487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.361061    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7315b844-4c65-4097-9656-ed99e582ce4c-istiod-ca-cert\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.369487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.361115    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-certs\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.369487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.361176    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-istio-token\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.369487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.362125    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.369487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.362372    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-data\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.387929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.387694    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7315b844-4c65-4097-9656-ed99e582ce4c-istiod-ca-cert\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.387929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.387881    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-certs\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.391101 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.391059    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-envoy\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.392793 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.392756    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7315b844-4c65-4097-9656-ed99e582ce4c-istio-podinfo\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.422947 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.422876    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-istio-token\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.423300 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.423282    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-kube-api-access-qdwfh\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.489504 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.489466    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:44.942206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.942178    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"]
Oct 31 19:06:44.943325 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:06:44.943299    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7315b844_4c65_4097_9656_ed99e582ce4c.slice/crio-985a85cd088636c79017dd0495f518e10465c94ef52723ada9269ae716b42c9e WatchSource:0}: Error finding container 985a85cd088636c79017dd0495f518e10465c94ef52723ada9269ae716b42c9e: Status 404 returned error can't find the container with id 985a85cd088636c79017dd0495f518e10465c94ef52723ada9269ae716b42c9e
Oct 31 19:06:44.945974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.945940    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:44.946058 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.946011    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:44.946058 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:44.946040    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:06:45.346936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:45.346844    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" event={"ID":"7315b844-4c65-4097-9656-ed99e582ce4c","Type":"ContainerStarted","Data":"9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0"}
Oct 31 19:06:45.346936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:45.346904    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" event={"ID":"7315b844-4c65-4097-9656-ed99e582ce4c","Type":"ContainerStarted","Data":"985a85cd088636c79017dd0495f518e10465c94ef52723ada9269ae716b42c9e"}
Oct 31 19:06:45.468470 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:45.465073    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" podStartSLOduration=1.465054388 podStartE2EDuration="1.465054388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:45.461744124 +0000 UTC m=+2708.692982415" watchObservedRunningTime="2026-10-31 19:06:45.465054388 +0000 UTC m=+2708.696292678"
Oct 31 19:06:45.490171 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:45.490141    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:46.496398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:46.496367    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:46.604693 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:06:46.604660    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:06:47.360100 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:47.360074    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:06:47.361181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:06:47.361161    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:07:04.968354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:04.968319    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"]
Oct 31 19:07:04.972719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:04.972696    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.043979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.043950    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-d1ff32-openshift-default-dockercfg-knnwq\""
Oct 31 19:07:05.063966 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.063916    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-credential-socket\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.064157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.063994    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-certs\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.064157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.064046    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebec53a-ceec-4521-8d79-b345aeefef4f-istiod-ca-cert\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.064157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.064103    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-token\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.064157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.064134    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfcj\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-kube-api-access-ssfcj\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.064381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.064221    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-envoy\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.064381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.064351    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-socket\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.064536 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.064392    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-podinfo\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.064536 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.064428    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-data\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.080561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.080525    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"]
Oct 31 19:07:05.166231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166181    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-data\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.166506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166258    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-credential-socket\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.166506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166382    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-certs\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.166506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166438    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebec53a-ceec-4521-8d79-b345aeefef4f-istiod-ca-cert\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.166708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166515    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-token\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.166708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166552    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfcj\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-kube-api-access-ssfcj\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.166708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166637    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-envoy\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.166919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166760    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-socket\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.166919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.166801    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-podinfo\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.168110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.168084    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebec53a-ceec-4521-8d79-b345aeefef4f-istiod-ca-cert\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.168572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.168546    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-data\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.169041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.168962    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-credential-socket\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.169236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.169211    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-certs\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.176318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.171778    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-socket\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.180789 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.180746    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-envoy\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.181007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.180988    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-podinfo\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.227145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.227071    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfcj\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-kube-api-access-ssfcj\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.227145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.227089    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-token\") pod \"e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") " pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.288520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.288489    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:05.719877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.719847    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"]
Oct 31 19:07:05.721207 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:07:05.721164    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebec53a_ceec_4521_8d79_b345aeefef4f.slice/crio-6f99cdd4ff4bc0593629b91aa94ce94e5715371e4123ce406f7f13dc65e032cb WatchSource:0}: Error finding container 6f99cdd4ff4bc0593629b91aa94ce94e5715371e4123ce406f7f13dc65e032cb: Status 404 returned error can't find the container with id 6f99cdd4ff4bc0593629b91aa94ce94e5715371e4123ce406f7f13dc65e032cb
Oct 31 19:07:05.724127 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.724086    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:07:05.724222 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.724168    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:07:05.724222 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:05.724205    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:07:06.534708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:06.534669    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" event={"ID":"7ebec53a-ceec-4521-8d79-b345aeefef4f","Type":"ContainerStarted","Data":"a79a0ed3f29b8343fe527ea0fa718b453c5ba51ccbe4011055dabf4f78cae591"}
Oct 31 19:07:06.535138 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:06.534735    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" event={"ID":"7ebec53a-ceec-4521-8d79-b345aeefef4f","Type":"ContainerStarted","Data":"6f99cdd4ff4bc0593629b91aa94ce94e5715371e4123ce406f7f13dc65e032cb"}
Oct 31 19:07:06.701932 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:06.701873    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" podStartSLOduration=2.701856518 podStartE2EDuration="2.701856518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:07:06.699988977 +0000 UTC m=+2729.931227268" watchObservedRunningTime="2026-10-31 19:07:06.701856518 +0000 UTC m=+2729.933094810"
Oct 31 19:07:07.288698 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:07.288671    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:07.294013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:07.293988    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:07.544514 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:07.544429    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:07.545451 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:07.545426    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:07:13.331656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.331590    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"]
Oct 31 19:07:13.359842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.359813    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.394933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.394486    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"]
Oct 31 19:07:13.506941 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.506905    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6279f5-openshift-default-dockercfg-d8cq6\""
Oct 31 19:07:13.532758 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.532709    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.532950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.532789    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.532950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.532825    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.532950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.532891    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcjsz\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-kube-api-access-lcjsz\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.533120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.533018    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.533120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.533080    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.533234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.533180    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.533234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.533226    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.533318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.533289    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.634706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.634677    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.634897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.634716    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.634897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.634785    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.634897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.634812    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.634897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.634841    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.634976    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.635053    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.635112    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635294 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.635155    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcjsz\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-kube-api-access-lcjsz\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635294 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.635235    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635294 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.635262    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.635562    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.635649    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.635841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.635825    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.637787 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.637771    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.638114 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.638090    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.692301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.692276    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.692420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.692393    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcjsz\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-kube-api-access-lcjsz\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:13.705143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:13.705116    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:14.145207 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:14.145173    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"]
Oct 31 19:07:14.201416 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:14.201367    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:07:14.201572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:14.201515    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:07:14.201653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:14.201579    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:07:14.633508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:14.631803    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" event={"ID":"e2b883f6-71bf-4f83-96d6-d75cb0532e0f","Type":"ContainerStarted","Data":"5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d"}
Oct 31 19:07:14.633508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:14.631930    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" event={"ID":"e2b883f6-71bf-4f83-96d6-d75cb0532e0f","Type":"ContainerStarted","Data":"0be729a446fb6e7ecbacf3f11af219033ae77ee1152d598592942d76e5594aef"}
Oct 31 19:07:14.705661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:14.705622    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:14.747444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:14.747256    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" podStartSLOduration=1.747237726 podStartE2EDuration="1.747237726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:07:14.74687619 +0000 UTC m=+2737.978114472" watchObservedRunningTime="2026-10-31 19:07:14.747237726 +0000 UTC m=+2737.978476020"
Oct 31 19:07:15.709166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:15.709048    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:07:15.709166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:15.709107    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 31 19:07:16.611213 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:07:16.611174    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:07:16.705811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:16.705779    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:07:16.706022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:16.705835    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 31 19:07:17.708896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:17.708861    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:07:17.709614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:17.709579    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 31 19:07:18.705710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:18.705676    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:07:18.705898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:18.705731    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 31 19:07:19.369489 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.367987    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"]
Oct 31 19:07:19.379738 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.379712    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.411379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.411343    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"]
Oct 31 19:07:19.440904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.440873    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-home\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.441100 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.441043    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-tmp-dir\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.441206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.441187    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b6ca1f-4d41-488f-8553-08a9f8703d45-tls-certs\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.441271 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.441244    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-dshm\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.441421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.441388    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbvm\" (UniqueName: \"kubernetes.io/projected/33b6ca1f-4d41-488f-8553-08a9f8703d45-kube-api-access-jdbvm\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.441603 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.441575    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-model-cache\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.474486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.468938    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a2c587\"/\"default-dockercfg-xj5xw\""
Oct 31 19:07:19.474486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.469071    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a2c587\"/\"praxis-route-a2c587-kserve-self-signed-certs\""
Oct 31 19:07:19.482237 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.482212    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a2c587\"/\"openshift-service-ca.crt\""
Oct 31 19:07:19.482370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.482252    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a2c587\"/\"kube-root-ca.crt\""
Oct 31 19:07:19.543315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.543282    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-dshm\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.543572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.543348    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbvm\" (UniqueName: \"kubernetes.io/projected/33b6ca1f-4d41-488f-8553-08a9f8703d45-kube-api-access-jdbvm\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.543572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.543418    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-model-cache\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.543572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.543484    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-home\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.543572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.543561    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-tmp-dir\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.543837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.543634    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b6ca1f-4d41-488f-8553-08a9f8703d45-tls-certs\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.544405 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.544380    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-model-cache\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.544540 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.544393    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-tmp-dir\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.544598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.544542    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-home\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.547010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.546968    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-dshm\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.547119 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.547028    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b6ca1f-4d41-488f-8553-08a9f8703d45-tls-certs\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.606660 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.606628    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbvm\" (UniqueName: \"kubernetes.io/projected/33b6ca1f-4d41-488f-8553-08a9f8703d45-kube-api-access-jdbvm\") pod \"praxis-route-a2c587-kserve-cb4b9497c-4s5ln\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") " pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.695205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.695175    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:19.706365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.706333    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:07:19.706559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:19.706391    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 31 19:07:20.126712 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:20.126682    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"]
Oct 31 19:07:20.127777 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:07:20.127744    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33b6ca1f_4d41_488f_8553_08a9f8703d45.slice/crio-662f96d964a14c034fd3907fd9ecce1f66c50c67b510ea092322f1a3ed22e4b8 WatchSource:0}: Error finding container 662f96d964a14c034fd3907fd9ecce1f66c50c67b510ea092322f1a3ed22e4b8: Status 404 returned error can't find the container with id 662f96d964a14c034fd3907fd9ecce1f66c50c67b510ea092322f1a3ed22e4b8
Oct 31 19:07:20.705586 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:20.705549    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:07:20.706086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:20.705596    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 31 19:07:20.712545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:20.712511    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln" event={"ID":"33b6ca1f-4d41-488f-8553-08a9f8703d45","Type":"ContainerStarted","Data":"82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be"}
Oct 31 19:07:20.712687 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:20.712560    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln" event={"ID":"33b6ca1f-4d41-488f-8553-08a9f8703d45","Type":"ContainerStarted","Data":"662f96d964a14c034fd3907fd9ecce1f66c50c67b510ea092322f1a3ed22e4b8"}
Oct 31 19:07:20.829097 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:20.829038    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln" podStartSLOduration=1.829021688 podStartE2EDuration="1.829021688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:07:20.826415869 +0000 UTC m=+2744.057654171" watchObservedRunningTime="2026-10-31 19:07:20.829021688 +0000 UTC m=+2744.060259979"
Oct 31 19:07:21.695355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:21.695317    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:21.992735 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:21.992700    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:21.993216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:21.993189    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:21.994138 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:21.994113    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:07:29.695828 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:29.695785    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:29.709155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:29.709126    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:29.803117 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:07:29.803082    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:07:46.606619 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:07:46.606571    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:08:16.609323 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:08:16.609243    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:08:18.401643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:18.401603    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:08:18.402041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:18.401672    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:08:20.267775 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:20.267739    5551 generic.go:356] "Generic (PLEG): container finished" podID="7315b844-4c65-4097-9656-ed99e582ce4c" containerID="9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0" exitCode=0
Oct 31 19:08:20.268205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:20.267807    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" event={"ID":"7315b844-4c65-4097-9656-ed99e582ce4c","Type":"ContainerDied","Data":"9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0"}
Oct 31 19:08:20.268302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:20.268285    5551 scope.go:122] "RemoveContainer" containerID="9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0"
Oct 31 19:08:20.270099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:20.270072    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:20.270184 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:20.270143    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:20.270184 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:20.270178    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:21.286295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:21.286255    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" event={"ID":"7315b844-4c65-4097-9656-ed99e582ce4c","Type":"ContainerStarted","Data":"81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458"}
Oct 31 19:08:21.495481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:21.492655    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:08:21.495481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:21.492850    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:08:21.495481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:21.492917    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 19:08:22.490722 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:22.490684    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:08:22.491129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:22.490746    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 19:08:23.490698 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:23.490654    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:08:23.490902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:23.490755    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 19:08:24.490248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:24.490206    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:08:24.490506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:24.490266    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 19:08:25.490336 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:25.490299    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:08:25.490874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:25.490389    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 19:08:26.494545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:26.494513    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:08:26.495129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:26.495095    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:08:26.495965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:26.495946    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:08:39.062242 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.062201    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"]
Oct 31 19:08:39.067259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.067234    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.122941 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.122908    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"]
Oct 31 19:08:39.167733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.167695    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-d8f34e\"/\"kube-root-ca.crt\""
Oct 31 19:08:39.167733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.167720    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-d8f34e\"/\"auth-test-model-d8f34e-kserve-self-signed-certs\""
Oct 31 19:08:39.167733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.167782    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-d8f34e\"/\"openshift-service-ca.crt\""
Oct 31 19:08:39.169017 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.168985    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-d8f34e\"/\"default-dockercfg-jzj4h\""
Oct 31 19:08:39.248627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.239721    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-dshm\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.248627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.239950    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tls-certs\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.248627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.240017    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-model-cache\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.248627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.240072    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpm64\" (UniqueName: \"kubernetes.io/projected/a7924fe1-e5b6-45ae-8814-ec2b596f5954-kube-api-access-cpm64\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.248627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.240126    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-home\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.248627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.240160    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tmp-dir\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342001    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpm64\" (UniqueName: \"kubernetes.io/projected/a7924fe1-e5b6-45ae-8814-ec2b596f5954-kube-api-access-cpm64\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342109    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-home\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342387 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342145    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tmp-dir\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342387 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342313    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-dshm\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342543 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342520    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tls-certs\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342589    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-model-cache\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342669 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342623    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-home\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342669 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342650    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tmp-dir\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.342992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.342966    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-model-cache\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.345497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.345474    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-dshm\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.345705 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.345688    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tls-certs\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.400614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.400577    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpm64\" (UniqueName: \"kubernetes.io/projected/a7924fe1-e5b6-45ae-8814-ec2b596f5954-kube-api-access-cpm64\") pod \"auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") " pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:39.687298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:39.687261    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:40.075188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:40.075091    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"]
Oct 31 19:08:40.431800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:40.431756    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5" event={"ID":"a7924fe1-e5b6-45ae-8814-ec2b596f5954","Type":"ContainerStarted","Data":"8ccabdad42c3c9a8ff8b4e4855bbd209a1f42e83c41502c7f645885fd3a12bb1"}
Oct 31 19:08:40.431800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:40.431808    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5" event={"ID":"a7924fe1-e5b6-45ae-8814-ec2b596f5954","Type":"ContainerStarted","Data":"a82116b177e51d8b32047b5c189e9a4a93628287937cbac7394bd2857786e049"}
Oct 31 19:08:40.546304 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:40.546095    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5" podStartSLOduration=1.546076768 podStartE2EDuration="1.546076768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:08:40.546011006 +0000 UTC m=+2823.777249297" watchObservedRunningTime="2026-10-31 19:08:40.546076768 +0000 UTC m=+2823.777315059"
Oct 31 19:08:40.688390 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:40.688305    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:42.448948 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:42.448915    5551 generic.go:356] "Generic (PLEG): container finished" podID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerID="a79a0ed3f29b8343fe527ea0fa718b453c5ba51ccbe4011055dabf4f78cae591" exitCode=0
Oct 31 19:08:42.449421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:42.448982    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" event={"ID":"7ebec53a-ceec-4521-8d79-b345aeefef4f","Type":"ContainerDied","Data":"a79a0ed3f29b8343fe527ea0fa718b453c5ba51ccbe4011055dabf4f78cae591"}
Oct 31 19:08:42.449656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:42.449640    5551 scope.go:122] "RemoveContainer" containerID="a79a0ed3f29b8343fe527ea0fa718b453c5ba51ccbe4011055dabf4f78cae591"
Oct 31 19:08:42.451452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:42.451417    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:42.451565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:42.451547    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:42.451614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:42.451600    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:43.482759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:43.482722    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" event={"ID":"7ebec53a-ceec-4521-8d79-b345aeefef4f","Type":"ContainerStarted","Data":"ba067a5aa0df8a18109e70562c6ff08f223f9da123147b4686761ad85be6318d"}
Oct 31 19:08:44.288669 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:44.288631    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:08:44.288928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:44.288904    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 19:08:44.288999 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:44.288957    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 19:08:45.292881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:45.292840    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 19:08:45.293559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:45.293525    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 19:08:46.292849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:46.292811    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 19:08:46.293204 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:46.293173    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 19:08:46.625361 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:08:46.625325    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:08:47.295484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:47.294851    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 19:08:47.295484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:47.294917    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 19:08:48.289489 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:48.289126    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 19:08:48.289489 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:48.289200    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 19:08:49.294276 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:49.294238    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:08:49.294810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:49.294791    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:08:49.295551 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:49.295527    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:08:49.687506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:49.687441    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:49.705310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:49.705280    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:50.637809 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:50.637777    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:08:54.661597 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:54.661562    5551 generic.go:356] "Generic (PLEG): container finished" podID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerID="e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0" exitCode=0
Oct 31 19:08:54.662149 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:54.661639    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" event={"ID":"d32f2799-c8ee-418d-ba27-31d90cba3a40","Type":"ContainerDied","Data":"e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0"}
Oct 31 19:08:54.662238 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:54.662168    5551 scope.go:122] "RemoveContainer" containerID="e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0"
Oct 31 19:08:54.663803 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:54.663777    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:54.663888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:54.663842    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:54.663888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:54.663876    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:08:55.687175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:55.687136    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" event={"ID":"d32f2799-c8ee-418d-ba27-31d90cba3a40","Type":"ContainerStarted","Data":"399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9"}
Oct 31 19:08:56.031848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:56.031757    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:08:56.033045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:56.033022    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:08:56.033139 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:56.033071    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:08:57.032424 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:57.032385    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:08:57.032922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:57.032485    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:08:58.032240 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:58.032203    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:08:58.032442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:58.032267    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:08:59.031837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:59.031807    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:08:59.032061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:59.031870    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:08:59.213849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:08:59.213802    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:09:00.032841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:00.032804    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:09:00.033024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:00.032856    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:09:00.140799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:00.140758    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"]
Oct 31 19:09:00.141102 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:00.141074    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy" containerID="cri-o://476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba" gracePeriod=30
Oct 31 19:09:00.181037 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:00.180986    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"]
Oct 31 19:09:00.181689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:00.181641    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy" containerID="cri-o://525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185" gracePeriod=30
Oct 31 19:09:00.190688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:00.190664    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"]
Oct 31 19:09:00.190983 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:00.190956    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy" containerID="cri-o://c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8" gracePeriod=30
Oct 31 19:09:01.038547 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:01.038516    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:09:01.039648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:01.039627    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:09:05.455725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.455640    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:09:05.525639 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.525613    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:09:05.528906 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.528884    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-kube-api-access-4zsvt\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-kube-api-access-4zsvt\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529039 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529022    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2facf719-8722-451b-8634-bfeeae70f02d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2facf719-8722-451b-8634-bfeeae70f02d-istiod-ca-cert\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529108 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529062    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-socket\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529108 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529100    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-certs\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529162    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2facf719-8722-451b-8634-bfeeae70f02d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2facf719-8722-451b-8634-bfeeae70f02d-istio-podinfo\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529279    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-credential-socket\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529320    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-data\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529415    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-envoy\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529512 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529481    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-istio-token\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-istio-token\") pod \"2facf719-8722-451b-8634-bfeeae70f02d\" (UID: \"2facf719-8722-451b-8634-bfeeae70f02d\") "
Oct 31 19:09:05.529575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529486    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-certs" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.529628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529564    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-credential-socket" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.529628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529591    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2facf719-8722-451b-8634-bfeeae70f02d-istiod-ca-cert" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:09:05.529741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529688    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-socket" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.529853 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.529835    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-data" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.532345 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.532318    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2facf719-8722-451b-8634-bfeeae70f02d-istio-podinfo" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:09:05.533301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533277    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-envoy" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.533411 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533315    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-istio-token" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:09:05.533584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533571    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.533633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533586    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.533633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533597    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2facf719-8722-451b-8634-bfeeae70f02d-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.533633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533607    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.533633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533616    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.533633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533625    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2facf719-8722-451b-8634-bfeeae70f02d-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.533633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533633    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.533829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.533641    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2facf719-8722-451b-8634-bfeeae70f02d-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.536061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.536043    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-kube-api-access-4zsvt" pod "2facf719-8722-451b-8634-bfeeae70f02d" (UID: "2facf719-8722-451b-8634-bfeeae70f02d"). InnerVolumeSpecName "kube-api-access-4zsvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:09:05.563692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.563667    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:09:05.634427 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634392    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-envoy\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.634619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634481    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-certs\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.634619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634509    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-token\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-token\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.634619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634563    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-socket\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.634811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634616    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-kube-api-access-b5bwm\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-kube-api-access-b5bwm\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.634811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634653    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-token\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-token\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.634811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634683    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-podinfo\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.634811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634720    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istiod-ca-cert\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.634811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634763    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-socket\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.635090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634938    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-credential-socket\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.635090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.634979    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-envoy\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.635090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635020    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-podinfo\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.635090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635087    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istiod-ca-cert\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.635308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635124    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-certs\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.635308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635166    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-kube-api-access-dl45t\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-kube-api-access-dl45t\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.635308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635187    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-certs" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.635308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635210    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-data\") pod \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\" (UID: \"c4aa41cd-e710-49da-bc0c-56ffb03609fd\") "
Oct 31 19:09:05.635308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635209    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-socket" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.635308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635247    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-credential-socket\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.635720 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635324    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-data\") pod \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\" (UID: \"741f42cf-57bd-455b-8fc5-7d0da0bd4682\") "
Oct 31 19:09:05.635720 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635560    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istiod-ca-cert" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:09:05.635929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.635905    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-data" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636311    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636333    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636383    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636401    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636417    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zsvt\" (UniqueName: \"kubernetes.io/projected/2facf719-8722-451b-8634-bfeeae70f02d-kube-api-access-4zsvt\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636480    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-data" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636636    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-socket" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636693    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-certs" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636828    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-credential-socket" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.637030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.636915    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-credential-socket" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.638322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.637053    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istiod-ca-cert" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:09:05.638322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.637990    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-envoy" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.638322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.638218    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-token" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:09:05.638836 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.638805    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-token" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:09:05.638930 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.638884    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-podinfo" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:09:05.639336 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.639311    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-envoy" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:05.640170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.640143    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-kube-api-access-b5bwm" pod "c4aa41cd-e710-49da-bc0c-56ffb03609fd" (UID: "c4aa41cd-e710-49da-bc0c-56ffb03609fd"). InnerVolumeSpecName "kube-api-access-b5bwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:09:05.640363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.640346    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-kube-api-access-dl45t" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "kube-api-access-dl45t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:09:05.640515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.640501    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-podinfo" pod "741f42cf-57bd-455b-8fc5-7d0da0bd4682" (UID: "741f42cf-57bd-455b-8fc5-7d0da0bd4682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:09:05.737358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737283    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737312    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737322    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737331    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737341    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl45t\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-kube-api-access-dl45t\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737351    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737359    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737367    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737376    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737385    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5bwm\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-kube-api-access-b5bwm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737393    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737410    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f42cf-57bd-455b-8fc5-7d0da0bd4682-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737421    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4aa41cd-e710-49da-bc0c-56ffb03609fd-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.737741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.737432    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f42cf-57bd-455b-8fc5-7d0da0bd4682-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:05.782954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.782920    5551 generic.go:356] "Generic (PLEG): container finished" podID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerID="c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8" exitCode=0
Oct 31 19:09:05.783098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.782970    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"
Oct 31 19:09:05.783098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.783007    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" event={"ID":"c4aa41cd-e710-49da-bc0c-56ffb03609fd","Type":"ContainerDied","Data":"c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8"}
Oct 31 19:09:05.783098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.783055    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd" event={"ID":"c4aa41cd-e710-49da-bc0c-56ffb03609fd","Type":"ContainerDied","Data":"5f50ad4c4da946a8eb2db3794ec92c0e44ff5ae49211041332c89adf8e2a9fbd"}
Oct 31 19:09:05.783098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.783079    5551 scope.go:122] "RemoveContainer" containerID="c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8"
Oct 31 19:09:05.784683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.784663    5551 generic.go:356] "Generic (PLEG): container finished" podID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerID="476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba" exitCode=0
Oct 31 19:09:05.784804 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.784703    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"
Oct 31 19:09:05.784880 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.784823    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" event={"ID":"741f42cf-57bd-455b-8fc5-7d0da0bd4682","Type":"ContainerDied","Data":"476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba"}
Oct 31 19:09:05.784942 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.784878    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt" event={"ID":"741f42cf-57bd-455b-8fc5-7d0da0bd4682","Type":"ContainerDied","Data":"398443ea72ab0ab5c77d71970eb614c9f515ea193d6dcd9e12a010b471e99f3d"}
Oct 31 19:09:05.786486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.786449    5551 generic.go:356] "Generic (PLEG): container finished" podID="2facf719-8722-451b-8634-bfeeae70f02d" containerID="525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185" exitCode=0
Oct 31 19:09:05.786616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.786532    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"
Oct 31 19:09:05.786616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.786563    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" event={"ID":"2facf719-8722-451b-8634-bfeeae70f02d","Type":"ContainerDied","Data":"525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185"}
Oct 31 19:09:05.786716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.786624    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d" event={"ID":"2facf719-8722-451b-8634-bfeeae70f02d","Type":"ContainerDied","Data":"e09aa2ebd98fd5aafbe5b5cda55e2a6ae12fd5ff82afa7ccc22c06fab747cad4"}
Oct 31 19:09:05.797785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.797763    5551 scope.go:122] "RemoveContainer" containerID="c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8"
Oct 31 19:09:05.798105 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:09:05.798085    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8\": container with ID starting with c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8 not found: ID does not exist" containerID="c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8"
Oct 31 19:09:05.798187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.798112    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8"} err="failed to get container status \"c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8\": rpc error: code = NotFound desc = could not find container \"c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8\": container with ID starting with c2d80c7ca86c710ff8ea5f04114703175df6ff73b0aeb54313ac00f4fb6d89f8 not found: ID does not exist"
Oct 31 19:09:05.798187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.798130    5551 scope.go:122] "RemoveContainer" containerID="476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba"
Oct 31 19:09:05.815701 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.815685    5551 scope.go:122] "RemoveContainer" containerID="476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba"
Oct 31 19:09:05.816088 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:09:05.816060    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba\": container with ID starting with 476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba not found: ID does not exist" containerID="476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba"
Oct 31 19:09:05.816221 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.816098    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba"} err="failed to get container status \"476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba\": rpc error: code = NotFound desc = could not find container \"476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba\": container with ID starting with 476fe52e992061a7780f965a095e5c708c597cd7eb932d608a75ebd7f17ea8ba not found: ID does not exist"
Oct 31 19:09:05.816221 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.816126    5551 scope.go:122] "RemoveContainer" containerID="525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185"
Oct 31 19:09:05.826787 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.826763    5551 scope.go:122] "RemoveContainer" containerID="525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185"
Oct 31 19:09:05.827081 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:09:05.827054    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185\": container with ID starting with 525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185 not found: ID does not exist" containerID="525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185"
Oct 31 19:09:05.827216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.827093    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185"} err="failed to get container status \"525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185\": rpc error: code = NotFound desc = could not find container \"525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185\": container with ID starting with 525758c3e9ec5fe049ddbea11b4aed3831ce06a9232e13e022a2ae08effab185 not found: ID does not exist"
Oct 31 19:09:05.928071 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.928036    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"]
Oct 31 19:09:05.932864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:05.932838    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-w8k6d"]
Oct 31 19:09:06.041194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:06.041101    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"]
Oct 31 19:09:06.046335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:06.046309    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-plcxt"]
Oct 31 19:09:06.154606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:06.154560    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"]
Oct 31 19:09:06.158056 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:06.158033    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-b2dcd"]
Oct 31 19:09:06.911569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:06.911536    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2facf719-8722-451b-8634-bfeeae70f02d" path="/var/lib/kubelet/pods/2facf719-8722-451b-8634-bfeeae70f02d/volumes"
Oct 31 19:09:06.912035 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:06.912021    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" path="/var/lib/kubelet/pods/741f42cf-57bd-455b-8fc5-7d0da0bd4682/volumes"
Oct 31 19:09:06.912434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:06.912422    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" path="/var/lib/kubelet/pods/c4aa41cd-e710-49da-bc0c-56ffb03609fd/volumes"
Oct 31 19:09:16.605776 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:09:16.605739    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:09:38.797827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.797787    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"]
Oct 31 19:09:38.798686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798667    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy"
Oct 31 19:09:38.798783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798687    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy"
Oct 31 19:09:38.798783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798742    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy"
Oct 31 19:09:38.798783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798750    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy"
Oct 31 19:09:38.798783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798767    5551 cpu_manager.go:414] "containerMap: removing container" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy"
Oct 31 19:09:38.798783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798774    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy"
Oct 31 19:09:38.799051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798894    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2facf719-8722-451b-8634-bfeeae70f02d" containerName="istio-proxy"
Oct 31 19:09:38.799051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798911    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4aa41cd-e710-49da-bc0c-56ffb03609fd" containerName="istio-proxy"
Oct 31 19:09:38.799051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.798923    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="741f42cf-57bd-455b-8fc5-7d0da0bd4682" containerName="istio-proxy"
Oct 31 19:09:38.801981 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.801959    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.854273 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.854241    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-1aefcd-openshift-default-dockercfg-xjbzs\""
Oct 31 19:09:38.877872 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.877838    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-certs\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.878033 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.877901    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-data\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.878033 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.878014    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-podinfo\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.878156 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.878135    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b869d2ac-8f69-4e85-8669-814ea8927d62-istiod-ca-cert\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.878199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.878187    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bjb\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-kube-api-access-59bjb\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.878290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.878273    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.878346 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.878331    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-credential-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.878446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.878426    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-token\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.878551 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.878509    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-envoy\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.911857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.911829    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"]
Oct 31 19:09:38.990485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990256    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-certs\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.990485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990341    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-data\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.990485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990380    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-podinfo\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.990814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990492    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b869d2ac-8f69-4e85-8669-814ea8927d62-istiod-ca-cert\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.990814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990543    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bjb\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-kube-api-access-59bjb\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.990814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990591    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.990814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990639    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-credential-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.990814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990673    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-token\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.990814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.990718    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-envoy\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.999565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.993333    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-credential-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.999565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.995867    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-certs\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:38.999565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:38.999518    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-data\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:39.045842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.045771    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:39.069591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.069514    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b869d2ac-8f69-4e85-8669-814ea8927d62-istiod-ca-cert\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:39.087772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.087701    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bjb\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-kube-api-access-59bjb\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:39.090656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.089450    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-envoy\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:39.090656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.090580    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-token\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:39.093010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.092987    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-podinfo\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:39.114691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.114660    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:39.571377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.571342    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"]
Oct 31 19:09:39.572650 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:09:39.572619    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb869d2ac_8f69_4e85_8669_814ea8927d62.slice/crio-ed5fe74903dfc48601f268b42c01d4fa220486187f587ee16ee334d228cbe78f WatchSource:0}: Error finding container ed5fe74903dfc48601f268b42c01d4fa220486187f587ee16ee334d228cbe78f: Status 404 returned error can't find the container with id ed5fe74903dfc48601f268b42c01d4fa220486187f587ee16ee334d228cbe78f
Oct 31 19:09:39.575211 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.575176    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:09:39.575302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.575244    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:09:39.575302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:39.575271    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:09:40.112113 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:40.112078    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" event={"ID":"b869d2ac-8f69-4e85-8669-814ea8927d62","Type":"ContainerStarted","Data":"3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5"}
Oct 31 19:09:40.112631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:40.112130    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" event={"ID":"b869d2ac-8f69-4e85-8669-814ea8927d62","Type":"ContainerStarted","Data":"ed5fe74903dfc48601f268b42c01d4fa220486187f587ee16ee334d228cbe78f"}
Oct 31 19:09:40.114836 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:40.114807    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:40.116955 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:40.116923    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 19:09:40.117103 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:40.116986    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 19:09:40.230134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:40.230071    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podStartSLOduration=2.230053263 podStartE2EDuration="2.230053263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:09:40.228548239 +0000 UTC m=+2883.459786529" watchObservedRunningTime="2026-10-31 19:09:40.230053263 +0000 UTC m=+2883.461291555"
Oct 31 19:09:41.120529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:41.120502    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:42.126473 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:42.126418    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:42.127384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:42.127365    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:09:45.223508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.223475    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"]
Oct 31 19:09:45.227423 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.227399    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.280029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.279993    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-home\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.280192 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.280079    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-model-cache\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.280192 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.280114    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tls-certs\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.280294 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.280276    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tmp-dir\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.280420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.280309    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vf9l\" (UniqueName: \"kubernetes.io/projected/a5a6cbdf-2115-4436-8cd4-1e45851105fb-kube-api-access-7vf9l\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.280420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.280337    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-dshm\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.331144 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.331095    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bbd8c6\"/\"openshift-service-ca.crt\""
Oct 31 19:09:45.331401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.331153    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bbd8c6\"/\"default-dockercfg-b75d9\""
Oct 31 19:09:45.331401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.331205    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bbd8c6\"/\"kube-root-ca.crt\""
Oct 31 19:09:45.331401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.331311    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bbd8c6\"/\"auth-test-model-bbd8c6-kserve-self-signed-certs\""
Oct 31 19:09:45.334888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.334865    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"]
Oct 31 19:09:45.381408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.381354    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tmp-dir\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.381637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.381433    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9l\" (UniqueName: \"kubernetes.io/projected/a5a6cbdf-2115-4436-8cd4-1e45851105fb-kube-api-access-7vf9l\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.381637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.381513    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-dshm\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.381637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.381593    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-home\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.381768 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.381661    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-model-cache\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.381768 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.381675    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tls-certs\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.381879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.381839    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tmp-dir\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.382029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.382009    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-home\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.382104 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.382088    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-model-cache\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.384247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.384217    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-dshm\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.384621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.384600    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tls-certs\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.438436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.438400    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vf9l\" (UniqueName: \"kubernetes.io/projected/a5a6cbdf-2115-4436-8cd4-1e45851105fb-kube-api-access-7vf9l\") pod \"auth-test-model-bbd8c6-kserve-85d9888b89-9v556\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") " pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.540417 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.540332    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:45.541794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.541772    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"]
Oct 31 19:09:45.918009 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:45.917985    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"]
Oct 31 19:09:45.919025 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:09:45.918995    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5a6cbdf_2115_4436_8cd4_1e45851105fb.slice/crio-898cb53daec594c1e0ee034e034f02ad262fc303762255154eeea84d1b449f86 WatchSource:0}: Error finding container 898cb53daec594c1e0ee034e034f02ad262fc303762255154eeea84d1b449f86: Status 404 returned error can't find the container with id 898cb53daec594c1e0ee034e034f02ad262fc303762255154eeea84d1b449f86
Oct 31 19:09:46.162982 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:46.162942    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556" event={"ID":"a5a6cbdf-2115-4436-8cd4-1e45851105fb","Type":"ContainerStarted","Data":"dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344"}
Oct 31 19:09:46.163185 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:46.163007    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556" event={"ID":"a5a6cbdf-2115-4436-8cd4-1e45851105fb","Type":"ContainerStarted","Data":"898cb53daec594c1e0ee034e034f02ad262fc303762255154eeea84d1b449f86"}
Oct 31 19:09:46.275943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:46.275840    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556" podStartSLOduration=1.27582467 podStartE2EDuration="1.27582467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:09:46.275219873 +0000 UTC m=+2889.506458186" watchObservedRunningTime="2026-10-31 19:09:46.27582467 +0000 UTC m=+2889.507062962"
Oct 31 19:09:46.540500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:46.540410    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:46.677766 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:09:46.677732    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:09:55.541034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:55.540993    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:55.555350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:55.555325    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:56.258284 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:56.258251    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:09:56.921151 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:56.921119    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"]
Oct 31 19:09:56.925270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:56.925251    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:56.981565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:56.981529    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"]
Oct 31 19:09:57.026947 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.026669    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-dbceb9\"/\"kube-root-ca.crt\""
Oct 31 19:09:57.026947 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.026755    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-dbceb9\"/\"openshift-service-ca.crt\""
Oct 31 19:09:57.026947 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.026917    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-dbceb9\"/\"default-dockercfg-cnps8\""
Oct 31 19:09:57.028700 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.028661    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-dbceb9\"/\"auth-test-model-dbceb9-kserve-self-signed-certs\""
Oct 31 19:09:57.042931 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.042903    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29jd\" (UniqueName: \"kubernetes.io/projected/da38213f-571c-48e0-b67a-bf85962ef952-kube-api-access-l29jd\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.043051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.043032    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-home\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.043123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.043101    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da38213f-571c-48e0-b67a-bf85962ef952-tls-certs\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.043188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.043165    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-model-cache\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.043329 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.043314    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-tmp-dir\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.043391 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.043350    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-dshm\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.144558 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.144442    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-tmp-dir\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.144778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.144614    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-dshm\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.144778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.144753    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29jd\" (UniqueName: \"kubernetes.io/projected/da38213f-571c-48e0-b67a-bf85962ef952-kube-api-access-l29jd\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.144986 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.144960    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-home\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.145088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.144994    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da38213f-571c-48e0-b67a-bf85962ef952-tls-certs\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.145088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.145046    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-model-cache\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.159641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.145540    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-model-cache\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.159641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.145809    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-tmp-dir\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.159641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.150764    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-home\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.171579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.170427    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-dshm\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.176806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.176766    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da38213f-571c-48e0-b67a-bf85962ef952-tls-certs\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.207354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.207325    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29jd\" (UniqueName: \"kubernetes.io/projected/da38213f-571c-48e0-b67a-bf85962ef952-kube-api-access-l29jd\") pod \"auth-test-model-dbceb9-kserve-65948c7989-qvvvv\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.239299 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.239259    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:09:57.707453 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:57.707422    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"]
Oct 31 19:09:57.708859 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:09:57.708830    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda38213f_571c_48e0_b67a_bf85962ef952.slice/crio-b8c5b509e61408002b3c4993d296d2004cfc7b60f28ef507ce9654214e02d09f WatchSource:0}: Error finding container b8c5b509e61408002b3c4993d296d2004cfc7b60f28ef507ce9654214e02d09f: Status 404 returned error can't find the container with id b8c5b509e61408002b3c4993d296d2004cfc7b60f28ef507ce9654214e02d09f
Oct 31 19:09:58.266085 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:58.266047    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv" event={"ID":"da38213f-571c-48e0-b67a-bf85962ef952","Type":"ContainerStarted","Data":"cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac"}
Oct 31 19:09:58.266636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:58.266103    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv" event={"ID":"da38213f-571c-48e0-b67a-bf85962ef952","Type":"ContainerStarted","Data":"b8c5b509e61408002b3c4993d296d2004cfc7b60f28ef507ce9654214e02d09f"}
Oct 31 19:09:58.382208 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:58.382153    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv" podStartSLOduration=2.382137289 podStartE2EDuration="2.382137289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:09:58.379804519 +0000 UTC m=+2901.611042813" watchObservedRunningTime="2026-10-31 19:09:58.382137289 +0000 UTC m=+2901.613375580"
Oct 31 19:09:59.239575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:09:59.239540    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:10:00.564412 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:00.564347    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln" podUID="33b6ca1f-4d41-488f-8553-08a9f8703d45" containerName="main" containerID="cri-o://82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be" gracePeriod=45
Oct 31 19:10:00.829350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:00.829294    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:10:01.001231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.001035    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b6ca1f-4d41-488f-8553-08a9f8703d45-kube-api-access-jdbvm\" (UniqueName: \"kubernetes.io/projected/33b6ca1f-4d41-488f-8553-08a9f8703d45-kube-api-access-jdbvm\") pod \"33b6ca1f-4d41-488f-8553-08a9f8703d45\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") "
Oct 31 19:10:01.001231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.001140    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-dshm\") pod \"33b6ca1f-4d41-488f-8553-08a9f8703d45\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") "
Oct 31 19:10:01.001542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.001240    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-home\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-home\") pod \"33b6ca1f-4d41-488f-8553-08a9f8703d45\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") "
Oct 31 19:10:01.001542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.001331    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-model-cache\") pod \"33b6ca1f-4d41-488f-8553-08a9f8703d45\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") "
Oct 31 19:10:01.001658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.001567    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-home" pod "33b6ca1f-4d41-488f-8553-08a9f8703d45" (UID: "33b6ca1f-4d41-488f-8553-08a9f8703d45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:01.001658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.001610    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-model-cache" pod "33b6ca1f-4d41-488f-8553-08a9f8703d45" (UID: "33b6ca1f-4d41-488f-8553-08a9f8703d45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:01.001751 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.001701    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-tmp-dir\") pod \"33b6ca1f-4d41-488f-8553-08a9f8703d45\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") "
Oct 31 19:10:01.001838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.001820    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b6ca1f-4d41-488f-8553-08a9f8703d45-tls-certs\" (UniqueName: \"kubernetes.io/secret/33b6ca1f-4d41-488f-8553-08a9f8703d45-tls-certs\") pod \"33b6ca1f-4d41-488f-8553-08a9f8703d45\" (UID: \"33b6ca1f-4d41-488f-8553-08a9f8703d45\") "
Oct 31 19:10:01.002042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.002017    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-tmp-dir" pod "33b6ca1f-4d41-488f-8553-08a9f8703d45" (UID: "33b6ca1f-4d41-488f-8553-08a9f8703d45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:01.002611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.002589    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:01.002611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.002612    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:01.002795 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.002626    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:01.005196 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.005169    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b6ca1f-4d41-488f-8553-08a9f8703d45-kube-api-access-jdbvm" pod "33b6ca1f-4d41-488f-8553-08a9f8703d45" (UID: "33b6ca1f-4d41-488f-8553-08a9f8703d45"). InnerVolumeSpecName "kube-api-access-jdbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:10:01.005398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.005376    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b6ca1f-4d41-488f-8553-08a9f8703d45-tls-certs" pod "33b6ca1f-4d41-488f-8553-08a9f8703d45" (UID: "33b6ca1f-4d41-488f-8553-08a9f8703d45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:10:01.006795 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.006774    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-dshm" pod "33b6ca1f-4d41-488f-8553-08a9f8703d45" (UID: "33b6ca1f-4d41-488f-8553-08a9f8703d45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:01.104059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.103970    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b6ca1f-4d41-488f-8553-08a9f8703d45-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:01.104059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.104000    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbvm\" (UniqueName: \"kubernetes.io/projected/33b6ca1f-4d41-488f-8553-08a9f8703d45-kube-api-access-jdbvm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:01.104059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.104017    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b6ca1f-4d41-488f-8553-08a9f8703d45-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:01.289769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.289731    5551 generic.go:356] "Generic (PLEG): container finished" podID="33b6ca1f-4d41-488f-8553-08a9f8703d45" containerID="82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be" exitCode=0
Oct 31 19:10:01.289978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.289791    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"
Oct 31 19:10:01.289978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.289818    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln" event={"ID":"33b6ca1f-4d41-488f-8553-08a9f8703d45","Type":"ContainerDied","Data":"82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be"}
Oct 31 19:10:01.289978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.289866    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln" event={"ID":"33b6ca1f-4d41-488f-8553-08a9f8703d45","Type":"ContainerDied","Data":"662f96d964a14c034fd3907fd9ecce1f66c50c67b510ea092322f1a3ed22e4b8"}
Oct 31 19:10:01.289978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.289893    5551 scope.go:122] "RemoveContainer" containerID="82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be"
Oct 31 19:10:01.302092 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.302064    5551 scope.go:122] "RemoveContainer" containerID="82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be"
Oct 31 19:10:01.302503 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:10:01.302441    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be\": container with ID starting with 82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be not found: ID does not exist" containerID="82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be"
Oct 31 19:10:01.302598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.302517    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be"} err="failed to get container status \"82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be\": rpc error: code = NotFound desc = could not find container \"82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be\": container with ID starting with 82b3634d493c565c8501cceedf091889aec0f655e2ab266a5866e994585db3be not found: ID does not exist"
Oct 31 19:10:01.453360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.453326    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"]
Oct 31 19:10:01.503160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.503118    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln"]
Oct 31 19:10:01.787070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:01.786971    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a2c587/praxis-route-a2c587-kserve-cb4b9497c-4s5ln" podUID="33b6ca1f-4d41-488f-8553-08a9f8703d45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:10:02.910093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:02.910054    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b6ca1f-4d41-488f-8553-08a9f8703d45" path="/var/lib/kubelet/pods/33b6ca1f-4d41-488f-8553-08a9f8703d45/volumes"
Oct 31 19:10:07.240338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:07.240307    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:10:07.254430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:07.254407    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:10:07.344512 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:07.344478    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:10:16.606336 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:10:16.606298    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:10:23.603994 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:23.603963    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"]
Oct 31 19:10:25.349394 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:25.349363    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"]
Oct 31 19:10:25.349951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:25.349614    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy" containerID="cri-o://60ef5af612eb242c64ec8c9661c1af04b662f6d9bcec9ef0fe9d2c3486fb432c" gracePeriod=30
Oct 31 19:10:28.605344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:28.605277    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988" podUID="5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" containerName="payload-pre-processing" containerID="cri-o://1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb" gracePeriod=25
Oct 31 19:10:29.488427 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.488390    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"]
Oct 31 19:10:29.489004 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.488990    5551 cpu_manager.go:414] "containerMap: removing container" podUID="33b6ca1f-4d41-488f-8553-08a9f8703d45" containerName="main"
Oct 31 19:10:29.489050 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.489006    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b6ca1f-4d41-488f-8553-08a9f8703d45" containerName="main"
Oct 31 19:10:29.489144 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.489135    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b6ca1f-4d41-488f-8553-08a9f8703d45" containerName="main"
Oct 31 19:10:29.494096 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.494076    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.549443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.549412    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"]
Oct 31 19:10:29.589148 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.589116    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2pw\" (UniqueName: \"kubernetes.io/projected/04aca74a-935f-46af-b988-925981022fcd-kube-api-access-lr2pw\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.589327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.589168    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-home\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.589327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.589238    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-dshm\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.589327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.589306    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-tmp-dir\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.589583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.589405    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04aca74a-935f-46af-b988-925981022fcd-tls-certs\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.589583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.589447    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-model-cache\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.594842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.594818    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c4731a\"/\"openshift-service-ca.crt\""
Oct 31 19:10:29.596125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.596108    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c4731a\"/\"default-dockercfg-bsqjx\""
Oct 31 19:10:29.596239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.596158    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c4731a\"/\"auth-test-model-c4731a-kserve-self-signed-certs\""
Oct 31 19:10:29.596239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.596233    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c4731a\"/\"kube-root-ca.crt\""
Oct 31 19:10:29.691281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691252    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-tmp-dir\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.691769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691323    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04aca74a-935f-46af-b988-925981022fcd-tls-certs\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.691769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691358    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-model-cache\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.691769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691543    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2pw\" (UniqueName: \"kubernetes.io/projected/04aca74a-935f-46af-b988-925981022fcd-kube-api-access-lr2pw\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.691769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691611    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-home\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.691769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691634    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-dshm\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.692025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691799    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-tmp-dir\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.692025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691840    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-model-cache\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.692025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.691893    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-home\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.693933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.693907    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-dshm\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.694312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.694293    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04aca74a-935f-46af-b988-925981022fcd-tls-certs\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.749344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.749279    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2pw\" (UniqueName: \"kubernetes.io/projected/04aca74a-935f-46af-b988-925981022fcd-kube-api-access-lr2pw\") pod \"auth-test-model-c4731a-kserve-64799c55bf-bfplm\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") " pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:29.807498 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:29.807448    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:30.191165 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.191130    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"]
Oct 31 19:10:30.191916 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:10:30.191888    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04aca74a_935f_46af_b988_925981022fcd.slice/crio-c8c4bcc8b3529bbc22141540dd2ede935ef2883c030569741040b9af356077f3 WatchSource:0}: Error finding container c8c4bcc8b3529bbc22141540dd2ede935ef2883c030569741040b9af356077f3: Status 404 returned error can't find the container with id c8c4bcc8b3529bbc22141540dd2ede935ef2883c030569741040b9af356077f3
Oct 31 19:10:30.551499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.548811    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm" event={"ID":"04aca74a-935f-46af-b988-925981022fcd","Type":"ContainerStarted","Data":"9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4"}
Oct 31 19:10:30.551499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.548866    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm" event={"ID":"04aca74a-935f-46af-b988-925981022fcd","Type":"ContainerStarted","Data":"c8c4bcc8b3529bbc22141540dd2ede935ef2883c030569741040b9af356077f3"}
Oct 31 19:10:30.553122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.553094    5551 generic.go:356] "Generic (PLEG): container finished" podID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerID="60ef5af612eb242c64ec8c9661c1af04b662f6d9bcec9ef0fe9d2c3486fb432c" exitCode=0
Oct 31 19:10:30.553248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.553179    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" event={"ID":"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5","Type":"ContainerDied","Data":"60ef5af612eb242c64ec8c9661c1af04b662f6d9bcec9ef0fe9d2c3486fb432c"}
Oct 31 19:10:30.600245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.600220    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:10:30.663781 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.663723    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm" podStartSLOduration=1.6637059760000001 podStartE2EDuration="1.663705976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:10:30.66309959 +0000 UTC m=+2933.894337881" watchObservedRunningTime="2026-10-31 19:10:30.663705976 +0000 UTC m=+2933.894944268"
Oct 31 19:10:30.712610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.712578    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istiod-ca-cert\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.712628    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-podinfo\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.712685    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-certs\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.712708    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-envoy\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.712730    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-credential-socket\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.712785    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-token\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-token\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713022    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-certs" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713055    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-kube-api-access-g9zgx\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-kube-api-access-g9zgx\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713098    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-socket\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713116    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-credential-socket" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:30.713135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713135    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-data\") pod \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\" (UID: \"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5\") "
Oct 31 19:10:30.713664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713053    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istiod-ca-cert" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:10:30.713664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713316    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-data" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:30.713842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713821    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-socket" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:30.714057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.713991    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:30.714057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.714036    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:30.714057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.714053    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:30.714057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.714067    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:30.714057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.714083    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:30.715385 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.715346    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-envoy" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:10:30.715729 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.715706    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-token" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:10:30.715920 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.715899    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-kube-api-access-g9zgx" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "kube-api-access-g9zgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:10:30.716020 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.715976    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-podinfo" pod "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" (UID: "8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:10:30.808016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.807978    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:30.816228 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.816195    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zgx\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-kube-api-access-g9zgx\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:30.816377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.816234    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:30.816377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.816255    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:30.816377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:30.816268    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:31.561916 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:31.561886    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"
Oct 31 19:10:31.561916 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:31.561898    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg" event={"ID":"8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5","Type":"ContainerDied","Data":"dbfc4a5207d6ccba9cca81f746a784f7d08b0f6136c3a73d15c527a99647dd02"}
Oct 31 19:10:31.562143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:31.561962    5551 scope.go:122] "RemoveContainer" containerID="60ef5af612eb242c64ec8c9661c1af04b662f6d9bcec9ef0fe9d2c3486fb432c"
Oct 31 19:10:31.727370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:31.727334    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"]
Oct 31 19:10:31.779114 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:31.779079    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-chhgg"]
Oct 31 19:10:32.909471 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:32.909432    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" path="/var/lib/kubelet/pods/8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5/volumes"
Oct 31 19:10:38.512968 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:38.512922    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988" podUID="5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:10:39.808022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:39.807985    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:39.822436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:39.822395    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:40.653923 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:40.653887    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:10:46.610110 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:10:46.610067    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:10:48.512795 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:48.512752    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988" podUID="5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:10:48.878119 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:48.878091    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:10:48.963648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:48.963621    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-kube-api-access-6xvhw\" (UniqueName: \"kubernetes.io/projected/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-kube-api-access-6xvhw\") pod \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\" (UID: \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\") "
Oct 31 19:10:48.963815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:48.963761    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-plugins-config-volume\") pod \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\" (UID: \"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7\") "
Oct 31 19:10:48.964081 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:48.964059    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-plugins-config-volume" pod "5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" (UID: "5a27c086-2d6e-4e13-ae84-fe5327ab8cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:10:48.964272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:48.964252    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:48.966070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:48.966048    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-kube-api-access-6xvhw" pod "5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" (UID: "5a27c086-2d6e-4e13-ae84-fe5327ab8cb7"). InnerVolumeSpecName "kube-api-access-6xvhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:10:49.066318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.066231    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xvhw\" (UniqueName: \"kubernetes.io/projected/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7-kube-api-access-6xvhw\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:49.727092 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.727058    5551 generic.go:356] "Generic (PLEG): container finished" podID="5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" containerID="1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb" exitCode=0
Oct 31 19:10:49.727619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.727118    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"
Oct 31 19:10:49.727619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.727148    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988" event={"ID":"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7","Type":"ContainerDied","Data":"1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb"}
Oct 31 19:10:49.727619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.727217    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988" event={"ID":"5a27c086-2d6e-4e13-ae84-fe5327ab8cb7","Type":"ContainerDied","Data":"ebc63b71883be9b8fc2447b4551e6ff09323368935f5d6c0106ec2e6bfdab5e0"}
Oct 31 19:10:49.727619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.727251    5551 scope.go:122] "RemoveContainer" containerID="1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb"
Oct 31 19:10:49.737653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.737599    5551 scope.go:122] "RemoveContainer" containerID="1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb"
Oct 31 19:10:49.737976 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:10:49.737950    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb\": container with ID starting with 1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb not found: ID does not exist" containerID="1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb"
Oct 31 19:10:49.738085 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.737987    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb"} err="failed to get container status \"1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb\": rpc error: code = NotFound desc = could not find container \"1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb\": container with ID starting with 1f863b5946cc32196511c97bc4ddfcce7ad0d56fe8c48c2a99245b6ad2c0aafb not found: ID does not exist"
Oct 31 19:10:49.892480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.892432    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"]
Oct 31 19:10:49.941670 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:49.941634    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2c587-78659f7ccf-r6988"]
Oct 31 19:10:50.910642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:10:50.910606    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" path="/var/lib/kubelet/pods/5a27c086-2d6e-4e13-ae84-fe5327ab8cb7/volumes"
Oct 31 19:11:12.518264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.518169    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"]
Oct 31 19:11:12.518998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.518968    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy"
Oct 31 19:11:12.518998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.518996    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy"
Oct 31 19:11:12.519195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.519043    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" containerName="payload-pre-processing"
Oct 31 19:11:12.519195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.519051    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" containerName="payload-pre-processing"
Oct 31 19:11:12.519305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.519229    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e01880f-5ba4-4e71-bcf5-7ef30d06c4d5" containerName="istio-proxy"
Oct 31 19:11:12.519305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.519261    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a27c086-2d6e-4e13-ae84-fe5327ab8cb7" containerName="payload-pre-processing"
Oct 31 19:11:12.522687 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.522667    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.591777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.588656    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d22e36\"/\"kube-root-ca.crt\""
Oct 31 19:11:12.592040 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.591986    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"]
Oct 31 19:11:12.598471 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.598401    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tls-certs\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.598616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.598592    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tmp-dir\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.598767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.598742    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-model-cache\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.598844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.598785    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlcs9\" (UniqueName: \"kubernetes.io/projected/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-kube-api-access-jlcs9\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.598915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.598892    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-home\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.598972 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.598936    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-dshm\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.631801 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.631777    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d22e36\"/\"openshift-service-ca.crt\""
Oct 31 19:11:12.631973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.631880    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d22e36\"/\"default-dockercfg-lhsh9\""
Oct 31 19:11:12.632039 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.631987    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d22e36\"/\"auth-test-model-d22e36-kserve-self-signed-certs\""
Oct 31 19:11:12.721487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.721379    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tmp-dir\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.723484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.722374    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tmp-dir\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.725343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.725006    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-model-cache\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.725343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.725047    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlcs9\" (UniqueName: \"kubernetes.io/projected/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-kube-api-access-jlcs9\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.725343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.725160    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-home\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.725343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.725199    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-dshm\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.725343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.725323    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tls-certs\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.725818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.725719    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-home\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.726026 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.725998    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-model-cache\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.729652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.728915    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-dshm\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.730500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.730478    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tls-certs\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.784025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.783949    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlcs9\" (UniqueName: \"kubernetes.io/projected/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-kube-api-access-jlcs9\") pod \"auth-test-model-d22e36-kserve-7656bbd65c-mqntv\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") " pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:12.835299 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:12.835260    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:13.277631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:13.277515    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"]
Oct 31 19:11:13.282987 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:11:13.282945    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfc3b0ee_40fb_491e_90c9_3403b12f9fc3.slice/crio-e5215739b3cf65ec49f8d9633d5b9ec8cbb69ca87e0e02a10d8b4caea802b67d WatchSource:0}: Error finding container e5215739b3cf65ec49f8d9633d5b9ec8cbb69ca87e0e02a10d8b4caea802b67d: Status 404 returned error can't find the container with id e5215739b3cf65ec49f8d9633d5b9ec8cbb69ca87e0e02a10d8b4caea802b67d
Oct 31 19:11:13.908197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:13.908160    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv" event={"ID":"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3","Type":"ContainerStarted","Data":"1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c"}
Oct 31 19:11:13.908197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:13.908201    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv" event={"ID":"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3","Type":"ContainerStarted","Data":"e5215739b3cf65ec49f8d9633d5b9ec8cbb69ca87e0e02a10d8b4caea802b67d"}
Oct 31 19:11:14.031042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:14.030985    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv" podStartSLOduration=2.030970871 podStartE2EDuration="2.030970871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:11:14.027425766 +0000 UTC m=+2977.258664086" watchObservedRunningTime="2026-10-31 19:11:14.030970871 +0000 UTC m=+2977.262209160"
Oct 31 19:11:14.835815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:14.835781    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:16.606111 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:11:16.606072    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:11:20.962304 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:20.962269    5551 generic.go:356] "Generic (PLEG): container finished" podID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerID="3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5" exitCode=0
Oct 31 19:11:20.962777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:20.962354    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" event={"ID":"b869d2ac-8f69-4e85-8669-814ea8927d62","Type":"ContainerDied","Data":"3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5"}
Oct 31 19:11:20.962943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:20.962923    5551 scope.go:122] "RemoveContainer" containerID="3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5"
Oct 31 19:11:20.964721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:20.964694    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:11:20.964843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:20.964761    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:11:20.964843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:20.964791    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:11:21.994734 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:21.988399    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" event={"ID":"b869d2ac-8f69-4e85-8669-814ea8927d62","Type":"ContainerStarted","Data":"8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12"}
Oct 31 19:11:22.115702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:22.115668    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:11:22.115877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:22.115843    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 19:11:22.115927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:22.115896    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 19:11:22.836286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:22.836253    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:22.849159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:22.849130    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:23.011438 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:23.011413    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:11:23.115120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:23.115035    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 19:11:23.115120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:23.115111    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 19:11:24.115719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:24.115675    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 19:11:24.116255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:24.115744    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 19:11:25.116024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:25.115980    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 19:11:25.116444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:25.116055    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 19:11:26.115204 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:26.115169    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 19:11:26.115386 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:26.115222    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 19:11:27.115545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:27.115505    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 19:11:27.115964 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:27.115576    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 19:11:27.126442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:27.126419    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:11:28.120068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:28.120036    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:11:28.121102 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:28.121085    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:11:35.186393 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.186353    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"]
Oct 31 19:11:35.186936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.186603    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp" podUID="0d366cb8-7755-423c-b137-92c62c614b43" containerName="maas-api" containerID="cri-o://c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6" gracePeriod=30
Oct 31 19:11:35.491200 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.491172    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:11:35.510272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.510243    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-metrics-certs\") pod \"0d366cb8-7755-423c-b137-92c62c614b43\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") "
Oct 31 19:11:35.510493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.510475    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d366cb8-7755-423c-b137-92c62c614b43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d366cb8-7755-423c-b137-92c62c614b43-openshift-service-ca\") pod \"0d366cb8-7755-423c-b137-92c62c614b43\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") "
Oct 31 19:11:35.510583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.510536    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-maas-api-tls\") pod \"0d366cb8-7755-423c-b137-92c62c614b43\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") "
Oct 31 19:11:35.510638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.510584    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d366cb8-7755-423c-b137-92c62c614b43-kube-api-access-mkmbn\" (UniqueName: \"kubernetes.io/projected/0d366cb8-7755-423c-b137-92c62c614b43-kube-api-access-mkmbn\") pod \"0d366cb8-7755-423c-b137-92c62c614b43\" (UID: \"0d366cb8-7755-423c-b137-92c62c614b43\") "
Oct 31 19:11:35.510821 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.510796    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d366cb8-7755-423c-b137-92c62c614b43-openshift-service-ca" pod "0d366cb8-7755-423c-b137-92c62c614b43" (UID: "0d366cb8-7755-423c-b137-92c62c614b43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:11:35.511281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.511260    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d366cb8-7755-423c-b137-92c62c614b43-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:35.512709 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.512681    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-metrics-certs" pod "0d366cb8-7755-423c-b137-92c62c614b43" (UID: "0d366cb8-7755-423c-b137-92c62c614b43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:11:35.512823 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.512724    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-maas-api-tls" pod "0d366cb8-7755-423c-b137-92c62c614b43" (UID: "0d366cb8-7755-423c-b137-92c62c614b43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:11:35.512823 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.512755    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d366cb8-7755-423c-b137-92c62c614b43-kube-api-access-mkmbn" pod "0d366cb8-7755-423c-b137-92c62c614b43" (UID: "0d366cb8-7755-423c-b137-92c62c614b43"). InnerVolumeSpecName "kube-api-access-mkmbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:11:35.616670 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.616632    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:35.616670 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.616673    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkmbn\" (UniqueName: \"kubernetes.io/projected/0d366cb8-7755-423c-b137-92c62c614b43-kube-api-access-mkmbn\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:35.616881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:35.616689    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d366cb8-7755-423c-b137-92c62c614b43-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:36.126097 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.126044    5551 generic.go:356] "Generic (PLEG): container finished" podID="0d366cb8-7755-423c-b137-92c62c614b43" containerID="c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6" exitCode=0
Oct 31 19:11:36.126651 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.126600    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp" event={"ID":"0d366cb8-7755-423c-b137-92c62c614b43","Type":"ContainerDied","Data":"c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6"}
Oct 31 19:11:36.126853 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.126828    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp" event={"ID":"0d366cb8-7755-423c-b137-92c62c614b43","Type":"ContainerDied","Data":"8ed64da670396789379cf95fc30575bbd617350302f7e818b7af3bc9a8ba89ef"}
Oct 31 19:11:36.126991 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.126977    5551 scope.go:122] "RemoveContainer" containerID="c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6"
Oct 31 19:11:36.127293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.127276    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"
Oct 31 19:11:36.226637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.226278    5551 scope.go:122] "RemoveContainer" containerID="c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6"
Oct 31 19:11:36.226891 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:11:36.226646    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6\": container with ID starting with c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6 not found: ID does not exist" containerID="c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6"
Oct 31 19:11:36.226891 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.226680    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6"} err="failed to get container status \"c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6\": rpc error: code = NotFound desc = could not find container \"c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6\": container with ID starting with c0b942698e8fc6cac4d79583023cf54e2819f0e7bf5f19409fcb048d799777a6 not found: ID does not exist"
Oct 31 19:11:36.317212 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.317174    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"]
Oct 31 19:11:36.317513 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.317495    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2c587-d4d885b7-2pcbp"]
Oct 31 19:11:36.911577 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:36.911544    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d366cb8-7755-423c-b137-92c62c614b43" path="/var/lib/kubelet/pods/0d366cb8-7755-423c-b137-92c62c614b43/volumes"
Oct 31 19:11:37.566030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:37.565999    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"]
Oct 31 19:11:37.566589 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:37.566219    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy" containerID="cri-o://0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f" gracePeriod=30
Oct 31 19:11:40.301693 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:40.301666    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:11:40.303703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:40.303671    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:11:40.686540 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:40.686498    5551 patch_prober.go:29] interesting pod/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:11:40.686757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:40.686567    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:11:42.839272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.839218    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:11:42.944888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.944860    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-kube-api-access-zwqvl\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-kube-api-access-zwqvl\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.944908    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-token\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-token\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.944937    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-credential-socket\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.944988    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-podinfo\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945058    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-envoy\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945139    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-data\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945201    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-socket\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945279    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-certs\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945300    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-credential-socket" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:42.945537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945338    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istiod-ca-cert\") pod \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\" (UID: \"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1\") "
Oct 31 19:11:42.945537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945498    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-data" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:42.945714 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945570    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-certs" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:42.945714 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.945583    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-socket" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:42.946057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.946023    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istiod-ca-cert" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:11:42.946210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.946187    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:42.946297 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.946218    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:42.946297 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.946234    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:42.946297 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.946250    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:42.946297 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.946265    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:42.947306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.947277    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-kube-api-access-zwqvl" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "kube-api-access-zwqvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:11:42.947427 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.947333    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-token" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:11:42.947597 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.947582    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-podinfo" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:11:42.947684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:42.947665    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-envoy" pod "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" (UID: "17ed5b8b-6ac5-423d-acdd-c067a0cf74f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:43.047206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.047173    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:43.047206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.047204    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqvl\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-kube-api-access-zwqvl\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:43.047521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.047216    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:43.047521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.047224    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:43.178649 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.178618    5551 generic.go:356] "Generic (PLEG): container finished" podID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerID="0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f" exitCode=0
Oct 31 19:11:43.178813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.178677    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"
Oct 31 19:11:43.178813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.178703    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" event={"ID":"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1","Type":"ContainerDied","Data":"0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f"}
Oct 31 19:11:43.178813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.178750    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb" event={"ID":"17ed5b8b-6ac5-423d-acdd-c067a0cf74f1","Type":"ContainerDied","Data":"f6e4dcabd0d93b96cc58865d619a9fea504292da8765b6d6883029032b91a200"}
Oct 31 19:11:43.178813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.178775    5551 scope.go:122] "RemoveContainer" containerID="0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f"
Oct 31 19:11:43.191745 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.191721    5551 scope.go:122] "RemoveContainer" containerID="a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5"
Oct 31 19:11:43.201838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.201825    5551 scope.go:122] "RemoveContainer" containerID="0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f"
Oct 31 19:11:43.202112 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:11:43.202092    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f\": container with ID starting with 0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f not found: ID does not exist" containerID="0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f"
Oct 31 19:11:43.202165 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.202119    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f"} err="failed to get container status \"0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f\": rpc error: code = NotFound desc = could not find container \"0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f\": container with ID starting with 0e41d860227e8e583195f096fc3e7f0fbd2e8aa215b93a748eb8702b99f6813f not found: ID does not exist"
Oct 31 19:11:43.202165 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.202135    5551 scope.go:122] "RemoveContainer" containerID="a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5"
Oct 31 19:11:43.202408 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:11:43.202389    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5\": container with ID starting with a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5 not found: ID does not exist" containerID="a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5"
Oct 31 19:11:43.202545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.202416    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5"} err="failed to get container status \"a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5\": rpc error: code = NotFound desc = could not find container \"a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5\": container with ID starting with a75c23493210b3ac688b44f88c221581e2fdafdf40a36ec9b9d256742d1a5dc5 not found: ID does not exist"
Oct 31 19:11:43.343877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.343841    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"]
Oct 31 19:11:43.347082 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:43.347046    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2c587-openshift-default-7cff4d8d85-cjdjb"]
Oct 31 19:11:44.910336 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:44.910304    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" path="/var/lib/kubelet/pods/17ed5b8b-6ac5-423d-acdd-c067a0cf74f1/volumes"
Oct 31 19:11:46.328645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:46.328613    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"]
Oct 31 19:11:46.329124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:46.328794    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy" containerID="cri-o://2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731" gracePeriod=30
Oct 31 19:11:46.670686 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:11:46.670641    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:11:50.732864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:50.732825    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"]
Oct 31 19:11:50.733277 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:50.733057    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h" podUID="a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" containerName="maas-api" containerID="cri-o://94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31" gracePeriod=30
Oct 31 19:11:51.029304 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.029270    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:11:51.066599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.063895    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-kube-api-access-sc6nf\" (UniqueName: \"kubernetes.io/projected/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-kube-api-access-sc6nf\") pod \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") "
Oct 31 19:11:51.066599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.064061    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls\") pod \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") "
Oct 31 19:11:51.066599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.064195    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-openshift-service-ca\") pod \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") "
Oct 31 19:11:51.066599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.064244    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-metrics-certs\") pod \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\" (UID: \"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b\") "
Oct 31 19:11:51.076930 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.076884    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-openshift-service-ca" pod "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" (UID: "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:11:51.077072 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.077034    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-metrics-certs" pod "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" (UID: "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:11:51.078822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.078791    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls" pod "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" (UID: "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:11:51.079224 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.079196    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-kube-api-access-sc6nf" pod "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" (UID: "a97b5e09-e843-4853-b7f6-8c1ac22d6f8b"). InnerVolumeSpecName "kube-api-access-sc6nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:11:51.188970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.188929    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.188970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.188968    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.189239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.188982    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.189239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.188996    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc6nf\" (UniqueName: \"kubernetes.io/projected/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b-kube-api-access-sc6nf\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.241218 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.241186    5551 generic.go:356] "Generic (PLEG): container finished" podID="a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" containerID="94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31" exitCode=0
Oct 31 19:11:51.241384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.241215    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h" event={"ID":"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b","Type":"ContainerDied","Data":"94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31"}
Oct 31 19:11:51.241384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.241244    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"
Oct 31 19:11:51.241384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.241295    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h" event={"ID":"a97b5e09-e843-4853-b7f6-8c1ac22d6f8b","Type":"ContainerDied","Data":"06e95589cf86c8864ddeaae8c1a46ac14bdffdc42c4a827fb22797465874bc14"}
Oct 31 19:11:51.241384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.241320    5551 scope.go:122] "RemoveContainer" containerID="94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31"
Oct 31 19:11:51.254647 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.252869    5551 scope.go:122] "RemoveContainer" containerID="94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31"
Oct 31 19:11:51.254647 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:11:51.253156    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31\": container with ID starting with 94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31 not found: ID does not exist" containerID="94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31"
Oct 31 19:11:51.254647 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.253190    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31"} err="failed to get container status \"94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31\": rpc error: code = NotFound desc = could not find container \"94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31\": container with ID starting with 94f967dc7de472fef696a6cdde9d63c8721e3f37007f8ed982b80279761a5e31 not found: ID does not exist"
Oct 31 19:11:51.407921 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.407894    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"]
Oct 31 19:11:51.411299 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.411275    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7006d3-b8cd4bbb6-f5k8h"]
Oct 31 19:11:51.580135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.580107    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713189    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-envoy\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713334    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-certs\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713408    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-kube-api-access-2b2xf\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-kube-api-access-2b2xf\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713489    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-data\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713519    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-credential-socket\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713553    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istiod-ca-cert\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713644    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-token\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-token\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713753    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-socket\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.714767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.713825    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-podinfo\") pod \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\" (UID: \"0946f9f5-25ab-4fc5-848f-1fc94017cdcf\") "
Oct 31 19:11:51.736782 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.736514    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-data" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.736782 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.736751    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-certs" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.736782 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.736767    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-socket" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.737350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.737219    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-podinfo" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:11:51.745630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.744640    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istiod-ca-cert" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:11:51.745948 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.745907    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-kube-api-access-2b2xf" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "kube-api-access-2b2xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:11:51.746841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.746818    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-credential-socket" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.769710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.767574    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-envoy" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.769859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.769705    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-token" pod "0946f9f5-25ab-4fc5-848f-1fc94017cdcf" (UID: "0946f9f5-25ab-4fc5-848f-1fc94017cdcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:11:51.816813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816763    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.816813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816813    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.817028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816828    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.817028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816840    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.817028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816851    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.817028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816862    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b2xf\" (UniqueName: \"kubernetes.io/projected/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-kube-api-access-2b2xf\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.817028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816877    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.817028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816889    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.817028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:51.816903    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0946f9f5-25ab-4fc5-848f-1fc94017cdcf-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:52.250616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.250583    5551 generic.go:356] "Generic (PLEG): container finished" podID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerID="2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731" exitCode=0
Oct 31 19:11:52.250838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.250637    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"
Oct 31 19:11:52.250838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.250657    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" event={"ID":"0946f9f5-25ab-4fc5-848f-1fc94017cdcf","Type":"ContainerDied","Data":"2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731"}
Oct 31 19:11:52.250838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.250728    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq" event={"ID":"0946f9f5-25ab-4fc5-848f-1fc94017cdcf","Type":"ContainerDied","Data":"49a075b3e2bd6cd3c87d1a40c2c4be4bd82c360b44af37271f28a3b68a8d21d8"}
Oct 31 19:11:52.250838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.250757    5551 scope.go:122] "RemoveContainer" containerID="2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731"
Oct 31 19:11:52.262685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.262668    5551 scope.go:122] "RemoveContainer" containerID="2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731"
Oct 31 19:11:52.263007 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:11:52.262983    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731\": container with ID starting with 2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731 not found: ID does not exist" containerID="2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731"
Oct 31 19:11:52.263064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.263016    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731"} err="failed to get container status \"2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731\": rpc error: code = NotFound desc = could not find container \"2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731\": container with ID starting with 2f63a746623ad1b447531cf5fd7d9d6cb9f025236f58cb02385055eee1093731 not found: ID does not exist"
Oct 31 19:11:52.415699 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.415670    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"]
Oct 31 19:11:52.465345 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.465316    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jh6bq"]
Oct 31 19:11:52.909178 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.909135    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" path="/var/lib/kubelet/pods/0946f9f5-25ab-4fc5-848f-1fc94017cdcf/volumes"
Oct 31 19:11:52.909865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:11:52.909844    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" path="/var/lib/kubelet/pods/a97b5e09-e843-4853-b7f6-8c1ac22d6f8b/volumes"
Oct 31 19:12:06.494389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.494355    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"]
Oct 31 19:12:06.494961 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.494946    5551 cpu_manager.go:414] "containerMap: removing container" podUID="a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" containerName="maas-api"
Oct 31 19:12:06.495011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.494963    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" containerName="maas-api"
Oct 31 19:12:06.495011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.494989    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy"
Oct 31 19:12:06.495011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.494994    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy"
Oct 31 19:12:06.495011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495004    5551 cpu_manager.go:414] "containerMap: removing container" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy"
Oct 31 19:12:06.495011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495009    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy"
Oct 31 19:12:06.495163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495022    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0d366cb8-7755-423c-b137-92c62c614b43" containerName="maas-api"
Oct 31 19:12:06.495163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495027    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d366cb8-7755-423c-b137-92c62c614b43" containerName="maas-api"
Oct 31 19:12:06.495163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495041    5551 cpu_manager.go:414] "containerMap: removing container" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy"
Oct 31 19:12:06.495163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495045    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy"
Oct 31 19:12:06.495163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495137    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0946f9f5-25ab-4fc5-848f-1fc94017cdcf" containerName="istio-proxy"
Oct 31 19:12:06.495163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495143    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy"
Oct 31 19:12:06.495163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495151    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d366cb8-7755-423c-b137-92c62c614b43" containerName="maas-api"
Oct 31 19:12:06.495163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495158    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ed5b8b-6ac5-423d-acdd-c067a0cf74f1" containerName="istio-proxy"
Oct 31 19:12:06.495406 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.495168    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97b5e09-e843-4853-b7f6-8c1ac22d6f8b" containerName="maas-api"
Oct 31 19:12:06.498429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.498414    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.556489 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.555637    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-d9922f-openshift-default-dockercfg-hw29l\""
Oct 31 19:12:06.606696 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.606641    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-certs\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.606896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.606737    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-credential-socket\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.606896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.606799    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96g6d\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-kube-api-access-96g6d\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.606896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.606860    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-token\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.606896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.606888    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-data\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.607131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.606958    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-envoy\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.607131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.606986    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-podinfo\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.607131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.607117    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istiod-ca-cert\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.607294 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.607143    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-socket\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.641530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.630139    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"]
Oct 31 19:12:06.708478 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708427    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-token\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.708691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708499    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-data\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.708691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708579    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-envoy\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.708691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708610    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-podinfo\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.708879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708739    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istiod-ca-cert\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.708879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708769    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-socket\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.708879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708846    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-certs\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.709067 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708923    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-credential-socket\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.709067 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.708981    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96g6d\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-kube-api-access-96g6d\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.710030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.709982    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-socket\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.710155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.710058    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-certs\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.710155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.710110    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-data\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.710624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.710555    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istiod-ca-cert\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.710624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.710636    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-credential-socket\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.712240 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.712219    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-envoy\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.712564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.712544    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-podinfo\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.768041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.767961    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96g6d\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-kube-api-access-96g6d\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.768163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.768048    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-token\") pod \"e2e-explicit-d9922f-openshift-default-86585f477c-xscph\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") " pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:06.811764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:06.811742    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:07.257381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:07.257352    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"]
Oct 31 19:12:07.258640 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:12:07.258617    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02a43a7_7a19_46a6_8b7d_b60666f9a615.slice/crio-ce213aaaec67b72e4c9b5be2f7ab8b9a2de90324bfdf69aa91cd927100d6d2b8 WatchSource:0}: Error finding container ce213aaaec67b72e4c9b5be2f7ab8b9a2de90324bfdf69aa91cd927100d6d2b8: Status 404 returned error can't find the container with id ce213aaaec67b72e4c9b5be2f7ab8b9a2de90324bfdf69aa91cd927100d6d2b8
Oct 31 19:12:07.261060 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:07.261025    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:12:07.261163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:07.261105    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:12:07.261163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:07.261156    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:12:07.361645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:07.361608    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" event={"ID":"b02a43a7-7a19-46a6-8b7d-b60666f9a615","Type":"ContainerStarted","Data":"884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8"}
Oct 31 19:12:07.361807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:07.361660    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" event={"ID":"b02a43a7-7a19-46a6-8b7d-b60666f9a615","Type":"ContainerStarted","Data":"ce213aaaec67b72e4c9b5be2f7ab8b9a2de90324bfdf69aa91cd927100d6d2b8"}
Oct 31 19:12:07.480299 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:07.480237    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" podStartSLOduration=1.480219463 podStartE2EDuration="1.480219463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:12:07.477304947 +0000 UTC m=+3030.708543233" watchObservedRunningTime="2026-10-31 19:12:07.480219463 +0000 UTC m=+3030.711457754"
Oct 31 19:12:07.813970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:07.813929    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:08.818890 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:08.818853    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:09.375524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:09.375476    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:09.376712 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:09.376691    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:12:16.604938 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:12:16.604896    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:12:17.223192 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.223159    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"]
Oct 31 19:12:17.227359 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.227343    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.253822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.253795    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-model-cache\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.253949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.253843    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-dshm\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.253949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.253905    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-home\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.254110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.254084    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352efcd-2ada-483a-8e2c-9c38aab157b1-tls-certs\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.254159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.254137    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mh7h\" (UniqueName: \"kubernetes.io/projected/0352efcd-2ada-483a-8e2c-9c38aab157b1-kube-api-access-8mh7h\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.254236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.254216    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-tmp-dir\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.329367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.329336    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6cfd68\"/\"kube-root-ca.crt\""
Oct 31 19:12:17.329584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.329528    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6cfd68\"/\"openshift-service-ca.crt\""
Oct 31 19:12:17.330051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.330029    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6cfd68\"/\"auth-test-model-6cfd68-kserve-self-signed-certs\""
Oct 31 19:12:17.330051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.330038    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6cfd68\"/\"default-dockercfg-4tflp\""
Oct 31 19:12:17.332886 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.332868    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"]
Oct 31 19:12:17.354978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.354951    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-home\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.355088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.355025    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352efcd-2ada-483a-8e2c-9c38aab157b1-tls-certs\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.355088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.355054    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mh7h\" (UniqueName: \"kubernetes.io/projected/0352efcd-2ada-483a-8e2c-9c38aab157b1-kube-api-access-8mh7h\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.355088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.355080    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-tmp-dir\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.355279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.355247    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-model-cache\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.355344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.355293    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-dshm\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.355420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.355397    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-home\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.355651 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.355631    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-model-cache\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.355868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.355824    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-tmp-dir\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.357935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.357912    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352efcd-2ada-483a-8e2c-9c38aab157b1-tls-certs\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.358208 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.358187    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-dshm\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.412815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.412783    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mh7h\" (UniqueName: \"kubernetes.io/projected/0352efcd-2ada-483a-8e2c-9c38aab157b1-kube-api-access-8mh7h\") pod \"auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.540728 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.540645    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:17.920904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:17.920875    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"]
Oct 31 19:12:17.922263 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:12:17.922229    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0352efcd_2ada_483a_8e2c_9c38aab157b1.slice/crio-7d1b72c7ccad9b24c449f5f3e9a2e8c56265510194c3c92f8973d265fd41eb50 WatchSource:0}: Error finding container 7d1b72c7ccad9b24c449f5f3e9a2e8c56265510194c3c92f8973d265fd41eb50: Status 404 returned error can't find the container with id 7d1b72c7ccad9b24c449f5f3e9a2e8c56265510194c3c92f8973d265fd41eb50
Oct 31 19:12:18.451120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:18.451080    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs" event={"ID":"0352efcd-2ada-483a-8e2c-9c38aab157b1","Type":"ContainerStarted","Data":"b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c"}
Oct 31 19:12:18.451347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:18.451130    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs" event={"ID":"0352efcd-2ada-483a-8e2c-9c38aab157b1","Type":"ContainerStarted","Data":"7d1b72c7ccad9b24c449f5f3e9a2e8c56265510194c3c92f8973d265fd41eb50"}
Oct 31 19:12:18.541129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:18.541094    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:18.571097 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:18.571048    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs" podStartSLOduration=1.5710336520000001 podStartE2EDuration="1.571033652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:12:18.567137548 +0000 UTC m=+3041.798375840" watchObservedRunningTime="2026-10-31 19:12:18.571033652 +0000 UTC m=+3041.802271942"
Oct 31 19:12:27.540926 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:27.540893    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:27.553572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:27.553545    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:28.535969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:28.535937    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:12:45.781440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:45.781352    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"]
Oct 31 19:12:45.781905 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:45.781579    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy" containerID="cri-o://5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d" gracePeriod=30
Oct 31 19:12:46.605045 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:12:46.605010    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:12:51.062918 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.062840    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:12:51.240314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240278    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-data\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.240539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240350    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-certs\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.240539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240403    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-envoy\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.240680 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240539    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-podinfo\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.240680 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240582    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-socket\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.240893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240670    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-data" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:51.240893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240682    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-certs" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:51.240893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240697    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istiod-ca-cert\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.240893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240749    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-kube-api-access-lcjsz\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-kube-api-access-lcjsz\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.240893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240802    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-credential-socket\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.240893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240841    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-socket" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:51.241270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.240913    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-token\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-token\") pod \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\" (UID: \"e2b883f6-71bf-4f83-96d6-d75cb0532e0f\") "
Oct 31 19:12:51.241270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.241061    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-credential-socket" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:51.241270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.241158    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istiod-ca-cert" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:12:51.241675 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.241598    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.241675 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.241624    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.241675 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.241649    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.241675 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.241664    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.241675 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.241678    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.243720 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.243692    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-token" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:12:51.243878 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.243858    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-envoy" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:51.244072 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.244046    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-podinfo" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:12:51.244156 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.244133    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-kube-api-access-lcjsz" pod "e2b883f6-71bf-4f83-96d6-d75cb0532e0f" (UID: "e2b883f6-71bf-4f83-96d6-d75cb0532e0f"). InnerVolumeSpecName "kube-api-access-lcjsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:12:51.343566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.343489    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.343566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.343517    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.343566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.343529    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcjsz\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-kube-api-access-lcjsz\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.343566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.343540    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b883f6-71bf-4f83-96d6-d75cb0532e0f-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:51.719011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.718980    5551 generic.go:356] "Generic (PLEG): container finished" podID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerID="5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d" exitCode=0
Oct 31 19:12:51.719190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.719021    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"
Oct 31 19:12:51.719190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.719066    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" event={"ID":"e2b883f6-71bf-4f83-96d6-d75cb0532e0f","Type":"ContainerDied","Data":"5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d"}
Oct 31 19:12:51.719190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.719126    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g" event={"ID":"e2b883f6-71bf-4f83-96d6-d75cb0532e0f","Type":"ContainerDied","Data":"0be729a446fb6e7ecbacf3f11af219033ae77ee1152d598592942d76e5594aef"}
Oct 31 19:12:51.719190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.719154    5551 scope.go:122] "RemoveContainer" containerID="5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d"
Oct 31 19:12:51.731072 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.731056    5551 scope.go:122] "RemoveContainer" containerID="5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d"
Oct 31 19:12:51.731351 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:12:51.731332    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d\": container with ID starting with 5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d not found: ID does not exist" containerID="5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d"
Oct 31 19:12:51.731391 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.731363    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d"} err="failed to get container status \"5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d\": rpc error: code = NotFound desc = could not find container \"5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d\": container with ID starting with 5644c503bed1ec9479fe15a646dab7762c1fef783a724336c485e62837d2b86d not found: ID does not exist"
Oct 31 19:12:51.884041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.884008    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"]
Oct 31 19:12:51.933153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:51.933121    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-8772g"]
Oct 31 19:12:52.909516 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:12:52.909486    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" path="/var/lib/kubelet/pods/e2b883f6-71bf-4f83-96d6-d75cb0532e0f/volumes"
Oct 31 19:13:16.604657 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:13:16.604621    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:13:46.606067 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:13:46.606030    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:13:50.169643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:50.169612    5551 generic.go:356] "Generic (PLEG): container finished" podID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerID="884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8" exitCode=0
Oct 31 19:13:50.170115 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:50.169682    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" event={"ID":"b02a43a7-7a19-46a6-8b7d-b60666f9a615","Type":"ContainerDied","Data":"884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8"}
Oct 31 19:13:50.170191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:50.170163    5551 scope.go:122] "RemoveContainer" containerID="884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8"
Oct 31 19:13:50.171947 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:50.171911    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:13:50.172074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:50.171985    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:13:50.172074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:50.172025    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:13:51.189931 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:51.189888    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" event={"ID":"b02a43a7-7a19-46a6-8b7d-b60666f9a615","Type":"ContainerStarted","Data":"da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062"}
Oct 31 19:13:51.812159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:51.812118    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:13:51.813491 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:51.813437    5551 patch_prober.go:29] interesting pod/e2e-explicit-d9922f-openshift-default-86585f477c-xscph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:13:51.813559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:51.813527    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 19:13:52.812322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:52.812284    5551 patch_prober.go:29] interesting pod/e2e-explicit-d9922f-openshift-default-86585f477c-xscph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:13:52.812873 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:52.812351    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 19:13:53.812843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:53.812806    5551 patch_prober.go:29] interesting pod/e2e-explicit-d9922f-openshift-default-86585f477c-xscph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:13:53.813270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:53.812859    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 19:13:54.378484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:54.377552    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:13:54.812380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:54.812285    5551 patch_prober.go:29] interesting pod/e2e-explicit-d9922f-openshift-default-86585f477c-xscph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:13:54.812380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:54.812358    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 19:13:55.813161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:55.813122    5551 patch_prober.go:29] interesting pod/e2e-explicit-d9922f-openshift-default-86585f477c-xscph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:13:55.813598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:55.813186    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 19:13:56.817304 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:56.817271    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:13:56.818321 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:13:56.818299    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:14:16.605222 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:14:16.605143    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:14:46.605993 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:14:46.605957    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:14:57.100554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:14:57.100525    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"]
Oct 31 19:14:57.101132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:14:57.100733    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy" containerID="cri-o://063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96" gracePeriod=30
Oct 31 19:15:00.179375 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.179310    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"]
Oct 31 19:15:00.180239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.180217    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy"
Oct 31 19:15:00.180239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.180240    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy"
Oct 31 19:15:00.180441 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.180426    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b883f6-71bf-4f83-96d6-d75cb0532e0f" containerName="istio-proxy"
Oct 31 19:15:00.184646 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.184622    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:00.188550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.188525    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"]
Oct 31 19:15:00.193868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.193095    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:00.193868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.193258    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"]
Oct 31 19:15:00.197925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.197904    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:00.214772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.214750    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"]
Oct 31 19:15:00.218678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.218660    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"]
Oct 31 19:15:00.218780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.218758    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:00.221335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.221309    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1819ada6-e365-4e02-9f20-79c8931eb80f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk\" (UID: \"1819ada6-e365-4e02-9f20-79c8931eb80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:00.221493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.221349    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4tz8\" (UniqueName: \"kubernetes.io/projected/56756066-3329-4ea4-8759-8dbe2f99fe16-kube-api-access-m4tz8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7\" (UID: \"56756066-3329-4ea4-8759-8dbe2f99fe16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:00.221493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.221365    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8qn\" (UniqueName: \"kubernetes.io/projected/1819ada6-e365-4e02-9f20-79c8931eb80f-kube-api-access-cc8qn\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk\" (UID: \"1819ada6-e365-4e02-9f20-79c8931eb80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:00.221645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.221583    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56756066-3329-4ea4-8759-8dbe2f99fe16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7\" (UID: \"56756066-3329-4ea4-8759-8dbe2f99fe16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:00.223001 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.222981    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:00.263615 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.263594    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"]
Oct 31 19:15:00.316369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.316338    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"]
Oct 31 19:15:00.320238 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.320212    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"]
Oct 31 19:15:00.321656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.321622    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:15:00.323362 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323338    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjl4\" (UniqueName: \"kubernetes.io/projected/b0cde556-93c8-440d-af71-4401046cc720-kube-api-access-lpjl4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87\" (UID: \"b0cde556-93c8-440d-af71-4401046cc720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:00.323512 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323391    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cde556-93c8-440d-af71-4401046cc720-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87\" (UID: \"b0cde556-93c8-440d-af71-4401046cc720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:00.323512 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323420    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbfc\" (UniqueName: \"kubernetes.io/projected/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-kube-api-access-dsbfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs\" (UID: \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:00.323664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323591    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cde15d-d21e-4c30-b17e-958e7e235910-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb\" (UID: \"47cde15d-d21e-4c30-b17e-958e7e235910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:00.323664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323633    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs\" (UID: \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:00.323780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323696    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56756066-3329-4ea4-8759-8dbe2f99fe16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7\" (UID: \"56756066-3329-4ea4-8759-8dbe2f99fe16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:00.323780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323759    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqz6d\" (UniqueName: \"kubernetes.io/projected/47cde15d-d21e-4c30-b17e-958e7e235910-kube-api-access-kqz6d\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb\" (UID: \"47cde15d-d21e-4c30-b17e-958e7e235910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:00.323898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323788    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1819ada6-e365-4e02-9f20-79c8931eb80f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk\" (UID: \"1819ada6-e365-4e02-9f20-79c8931eb80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:00.323898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323815    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4tz8\" (UniqueName: \"kubernetes.io/projected/56756066-3329-4ea4-8759-8dbe2f99fe16-kube-api-access-m4tz8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7\" (UID: \"56756066-3329-4ea4-8759-8dbe2f99fe16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:00.323898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.323838    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8qn\" (UniqueName: \"kubernetes.io/projected/1819ada6-e365-4e02-9f20-79c8931eb80f-kube-api-access-cc8qn\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk\" (UID: \"1819ada6-e365-4e02-9f20-79c8931eb80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:00.324495 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.324434    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1819ada6-e365-4e02-9f20-79c8931eb80f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk\" (UID: \"1819ada6-e365-4e02-9f20-79c8931eb80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:00.324712 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.324687    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56756066-3329-4ea4-8759-8dbe2f99fe16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7\" (UID: \"56756066-3329-4ea4-8759-8dbe2f99fe16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:00.375419 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.375389    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"]
Oct 31 19:15:00.379818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.379791    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"]
Oct 31 19:15:00.381757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.381450    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4tz8\" (UniqueName: \"kubernetes.io/projected/56756066-3329-4ea4-8759-8dbe2f99fe16-kube-api-access-m4tz8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7\" (UID: \"56756066-3329-4ea4-8759-8dbe2f99fe16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:00.389902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.388529    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8qn\" (UniqueName: \"kubernetes.io/projected/1819ada6-e365-4e02-9f20-79c8931eb80f-kube-api-access-cc8qn\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk\" (UID: \"1819ada6-e365-4e02-9f20-79c8931eb80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:00.424776 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.424747    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjl4\" (UniqueName: \"kubernetes.io/projected/b0cde556-93c8-440d-af71-4401046cc720-kube-api-access-lpjl4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87\" (UID: \"b0cde556-93c8-440d-af71-4401046cc720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:00.424927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.424826    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cde556-93c8-440d-af71-4401046cc720-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87\" (UID: \"b0cde556-93c8-440d-af71-4401046cc720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:00.424927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.424865    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbfc\" (UniqueName: \"kubernetes.io/projected/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-kube-api-access-dsbfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs\" (UID: \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:00.425066 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.424999    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cde15d-d21e-4c30-b17e-958e7e235910-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb\" (UID: \"47cde15d-d21e-4c30-b17e-958e7e235910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:00.425066 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.425048    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs\" (UID: \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:00.425200 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.425183    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz6d\" (UniqueName: \"kubernetes.io/projected/47cde15d-d21e-4c30-b17e-958e7e235910-kube-api-access-kqz6d\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb\" (UID: \"47cde15d-d21e-4c30-b17e-958e7e235910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:00.425570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.425553    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cde556-93c8-440d-af71-4401046cc720-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87\" (UID: \"b0cde556-93c8-440d-af71-4401046cc720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:00.425665 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.425648    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs\" (UID: \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:00.425706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.425657    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cde15d-d21e-4c30-b17e-958e7e235910-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb\" (UID: \"47cde15d-d21e-4c30-b17e-958e7e235910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:00.481181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.481104    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbfc\" (UniqueName: \"kubernetes.io/projected/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-kube-api-access-dsbfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs\" (UID: \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:00.481181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.481125    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjl4\" (UniqueName: \"kubernetes.io/projected/b0cde556-93c8-440d-af71-4401046cc720-kube-api-access-lpjl4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87\" (UID: \"b0cde556-93c8-440d-af71-4401046cc720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:00.481575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.481553    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqz6d\" (UniqueName: \"kubernetes.io/projected/47cde15d-d21e-4c30-b17e-958e7e235910-kube-api-access-kqz6d\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb\" (UID: \"47cde15d-d21e-4c30-b17e-958e7e235910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:00.502293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.502264    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:00.511714 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.511611    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:00.517684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.517662    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:00.533589 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.533568    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:00.540413 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.540331    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:00.939884 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.939853    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"]
Oct 31 19:15:00.941915 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:15:00.941888    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1819ada6_e365_4e02_9f20_79c8931eb80f.slice/crio-8523a7072d5babc10e2bcbba9c0374cac1e069be184b5144d080ae6132ee7cb8 WatchSource:0}: Error finding container 8523a7072d5babc10e2bcbba9c0374cac1e069be184b5144d080ae6132ee7cb8: Status 404 returned error can't find the container with id 8523a7072d5babc10e2bcbba9c0374cac1e069be184b5144d080ae6132ee7cb8
Oct 31 19:15:00.987865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.983909    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"]
Oct 31 19:15:00.993661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.993585    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"]
Oct 31 19:15:00.997055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:00.997036    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"]
Oct 31 19:15:01.001221 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.000554    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"]
Oct 31 19:15:01.727360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.727325    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" event={"ID":"56756066-3329-4ea4-8759-8dbe2f99fe16","Type":"ContainerStarted","Data":"34e3a53502ba5e47dc45aeb27cdfdcee4834c78bb8e91c39eb699533f616a663"}
Oct 31 19:15:01.727833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.727379    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" event={"ID":"56756066-3329-4ea4-8759-8dbe2f99fe16","Type":"ContainerStarted","Data":"845a7f27813dac3dc28584c9086bf5ef74d6bf786adda1c7b8d07545b32823ba"}
Oct 31 19:15:01.729136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.729112    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" event={"ID":"1819ada6-e365-4e02-9f20-79c8931eb80f","Type":"ContainerStarted","Data":"7481f88194b94520d70bd11f0f7052adf550aef56c26e2e166806bb1e08d85b8"}
Oct 31 19:15:01.729264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.729157    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" event={"ID":"1819ada6-e365-4e02-9f20-79c8931eb80f","Type":"ContainerStarted","Data":"8523a7072d5babc10e2bcbba9c0374cac1e069be184b5144d080ae6132ee7cb8"}
Oct 31 19:15:01.730733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.730712    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" event={"ID":"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6","Type":"ContainerStarted","Data":"9820c029ecd4913ddcdff41ae1e8ebada8ff8b918e88c8f5688a2b564afe7104"}
Oct 31 19:15:01.730824 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.730747    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" event={"ID":"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6","Type":"ContainerStarted","Data":"5df0838150ec8817d25ea35dcfc6268f1b02ee9e598dad6589a55b893758ef99"}
Oct 31 19:15:01.732193 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.732167    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" event={"ID":"47cde15d-d21e-4c30-b17e-958e7e235910","Type":"ContainerStarted","Data":"7b61173fdbfa5b90ee9e95ce7adf235167ec89378a0883596a2c3db4267c6793"}
Oct 31 19:15:01.732301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.732212    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" event={"ID":"47cde15d-d21e-4c30-b17e-958e7e235910","Type":"ContainerStarted","Data":"a385e5b5add16ee940d7bdc255477f4d3cead63d84f8965c5a418ca156e579f2"}
Oct 31 19:15:01.733755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.733736    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" event={"ID":"b0cde556-93c8-440d-af71-4401046cc720","Type":"ContainerStarted","Data":"eef0ac8f989d14931ab03b636b91d540b7077dcd7740b370119ed7bbf739a9d3"}
Oct 31 19:15:01.733858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.733765    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" event={"ID":"b0cde556-93c8-440d-af71-4401046cc720","Type":"ContainerStarted","Data":"c4ba5032a1e8e19a3bb0ae3770d2936be89fc7e64413e781577a63d9db51bdf1"}
Oct 31 19:15:01.841364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.841263    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" podStartSLOduration=1.84124954 podStartE2EDuration="1.84124954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:01.838069932 +0000 UTC m=+3205.069308225" watchObservedRunningTime="2026-10-31 19:15:01.84124954 +0000 UTC m=+3205.072487829"
Oct 31 19:15:01.958733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:01.958309    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" podStartSLOduration=1.958289775 podStartE2EDuration="1.958289775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:01.957491649 +0000 UTC m=+3205.188729940" watchObservedRunningTime="2026-10-31 19:15:01.958289775 +0000 UTC m=+3205.189528067"
Oct 31 19:15:02.071975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.071910    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" podStartSLOduration=2.071893104 podStartE2EDuration="2.071893104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:02.069610839 +0000 UTC m=+3205.300849130" watchObservedRunningTime="2026-10-31 19:15:02.071893104 +0000 UTC m=+3205.303131396"
Oct 31 19:15:02.186491 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.186426    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" podStartSLOduration=2.186411089 podStartE2EDuration="2.186411089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:02.182018036 +0000 UTC m=+3205.413256335" watchObservedRunningTime="2026-10-31 19:15:02.186411089 +0000 UTC m=+3205.417649379"
Oct 31 19:15:02.300671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.300614    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" podStartSLOduration=2.300594935 podStartE2EDuration="2.300594935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:02.298153493 +0000 UTC m=+3205.529391783" watchObservedRunningTime="2026-10-31 19:15:02.300594935 +0000 UTC m=+3205.531833226"
Oct 31 19:15:02.444190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.444157    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:15:02.556774 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.556746    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-socket\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.556960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.556795    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istiod-ca-cert\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.556960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.556843    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-certs\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.556960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.556924    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-data\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.556960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.556941    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-credential-socket\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.557199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.556981    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-token\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-token\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.557199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557030    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-envoy\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.557199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557052    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-socket" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:02.557199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557134    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-podinfo\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.557199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557182    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-kube-api-access-ljx64\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-kube-api-access-ljx64\") pod \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\" (UID: \"d0bff25e-b2ff-42a7-bcff-c559781d1d9d\") "
Oct 31 19:15:02.557520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557223    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-certs" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:02.557520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557241    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istiod-ca-cert" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:02.557520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557435    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-data" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:02.557742 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557718    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-credential-socket" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:02.557940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557830    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.557940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557845    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.557940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557855    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.557940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557864    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.557940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.557873    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.559927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.559902    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-podinfo" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:15:02.560022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.559928    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-envoy" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:02.560246 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.560222    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-kube-api-access-ljx64" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "kube-api-access-ljx64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:02.560593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.560571    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-token" pod "d0bff25e-b2ff-42a7-bcff-c559781d1d9d" (UID: "d0bff25e-b2ff-42a7-bcff-c559781d1d9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:02.659739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.659710    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.659739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.659737    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljx64\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-kube-api-access-ljx64\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.659991 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.659747    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.659991 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.659757    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0bff25e-b2ff-42a7-bcff-c559781d1d9d-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:02.742839 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.742807    5551 generic.go:356] "Generic (PLEG): container finished" podID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerID="9820c029ecd4913ddcdff41ae1e8ebada8ff8b918e88c8f5688a2b564afe7104" exitCode=60
Oct 31 19:15:02.743301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.742888    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" event={"ID":"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6","Type":"ContainerDied","Data":"9820c029ecd4913ddcdff41ae1e8ebada8ff8b918e88c8f5688a2b564afe7104"}
Oct 31 19:15:02.743301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.743137    5551 scope.go:122] "RemoveContainer" containerID="9820c029ecd4913ddcdff41ae1e8ebada8ff8b918e88c8f5688a2b564afe7104"
Oct 31 19:15:02.744790 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.744756    5551 generic.go:356] "Generic (PLEG): container finished" podID="47cde15d-d21e-4c30-b17e-958e7e235910" containerID="7b61173fdbfa5b90ee9e95ce7adf235167ec89378a0883596a2c3db4267c6793" exitCode=60
Oct 31 19:15:02.744914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.744786    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" event={"ID":"47cde15d-d21e-4c30-b17e-958e7e235910","Type":"ContainerDied","Data":"7b61173fdbfa5b90ee9e95ce7adf235167ec89378a0883596a2c3db4267c6793"}
Oct 31 19:15:02.745086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.745070    5551 scope.go:122] "RemoveContainer" containerID="7b61173fdbfa5b90ee9e95ce7adf235167ec89378a0883596a2c3db4267c6793"
Oct 31 19:15:02.751511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.751380    5551 generic.go:356] "Generic (PLEG): container finished" podID="b0cde556-93c8-440d-af71-4401046cc720" containerID="eef0ac8f989d14931ab03b636b91d540b7077dcd7740b370119ed7bbf739a9d3" exitCode=60
Oct 31 19:15:02.751511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.751476    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" event={"ID":"b0cde556-93c8-440d-af71-4401046cc720","Type":"ContainerDied","Data":"eef0ac8f989d14931ab03b636b91d540b7077dcd7740b370119ed7bbf739a9d3"}
Oct 31 19:15:02.751718 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.751702    5551 scope.go:122] "RemoveContainer" containerID="eef0ac8f989d14931ab03b636b91d540b7077dcd7740b370119ed7bbf739a9d3"
Oct 31 19:15:02.753436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.753417    5551 generic.go:356] "Generic (PLEG): container finished" podID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerID="063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96" exitCode=0
Oct 31 19:15:02.753570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.753506    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"
Oct 31 19:15:02.753637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.753569    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" event={"ID":"d0bff25e-b2ff-42a7-bcff-c559781d1d9d","Type":"ContainerDied","Data":"063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96"}
Oct 31 19:15:02.753702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.753682    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" event={"ID":"d0bff25e-b2ff-42a7-bcff-c559781d1d9d","Type":"ContainerDied","Data":"3b6e88cbbc9483a6cb19a7b2a24718e57d42d95d607ef6d39c0ef19e735f2a00"}
Oct 31 19:15:02.753751 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.753709    5551 scope.go:122] "RemoveContainer" containerID="063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96"
Oct 31 19:15:02.755395 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.755373    5551 generic.go:356] "Generic (PLEG): container finished" podID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerID="34e3a53502ba5e47dc45aeb27cdfdcee4834c78bb8e91c39eb699533f616a663" exitCode=60
Oct 31 19:15:02.755521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.755497    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" event={"ID":"56756066-3329-4ea4-8759-8dbe2f99fe16","Type":"ContainerDied","Data":"34e3a53502ba5e47dc45aeb27cdfdcee4834c78bb8e91c39eb699533f616a663"}
Oct 31 19:15:02.756743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.756727    5551 scope.go:122] "RemoveContainer" containerID="34e3a53502ba5e47dc45aeb27cdfdcee4834c78bb8e91c39eb699533f616a663"
Oct 31 19:15:02.768146 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.768120    5551 generic.go:356] "Generic (PLEG): container finished" podID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerID="7481f88194b94520d70bd11f0f7052adf550aef56c26e2e166806bb1e08d85b8" exitCode=60
Oct 31 19:15:02.768262 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.768160    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" event={"ID":"1819ada6-e365-4e02-9f20-79c8931eb80f","Type":"ContainerDied","Data":"7481f88194b94520d70bd11f0f7052adf550aef56c26e2e166806bb1e08d85b8"}
Oct 31 19:15:02.768521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.768503    5551 scope.go:122] "RemoveContainer" containerID="7481f88194b94520d70bd11f0f7052adf550aef56c26e2e166806bb1e08d85b8"
Oct 31 19:15:02.773900 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.773816    5551 scope.go:122] "RemoveContainer" containerID="063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96"
Oct 31 19:15:02.774142 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:02.774114    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96\": container with ID starting with 063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96 not found: ID does not exist" containerID="063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96"
Oct 31 19:15:02.774214 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:02.774156    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96"} err="failed to get container status \"063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96\": rpc error: code = NotFound desc = could not find container \"063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96\": container with ID starting with 063a5c474e27883a598f6e51f8b7222a87e202051ee0cc1d9968a6be2d64ee96 not found: ID does not exist"
Oct 31 19:15:03.298699 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.298661    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:15:03.298904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.298730    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:15:03.430477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.430428    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"]
Oct 31 19:15:03.436511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.436481    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-wwxzc"]
Oct 31 19:15:03.790287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.790251    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" event={"ID":"56756066-3329-4ea4-8759-8dbe2f99fe16","Type":"ContainerStarted","Data":"baa76c6eb5893092d25c2345cd29b3f533af88def947c30fcc66a6d313e90553"}
Oct 31 19:15:03.794375 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.794347    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" event={"ID":"1819ada6-e365-4e02-9f20-79c8931eb80f","Type":"ContainerStarted","Data":"b765254457cb58264d4c3ae070b98612a71b69c31cbf7728850e224bec32419e"}
Oct 31 19:15:03.797587 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.797563    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" event={"ID":"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6","Type":"ContainerStarted","Data":"983a2a8fefca644e98f93aaf89f2fed0d50fc8a7bfbdf83792a8585ca6539ead"}
Oct 31 19:15:03.800527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.800491    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" event={"ID":"47cde15d-d21e-4c30-b17e-958e7e235910","Type":"ContainerStarted","Data":"35e1bef7737d63ce6ce88a698952abf4ee59cf88a6473faebdcd0f45f40cfbe8"}
Oct 31 19:15:03.805680 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:03.805648    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" event={"ID":"b0cde556-93c8-440d-af71-4401046cc720","Type":"ContainerStarted","Data":"b93abc7425c42ff46d82e3f2e8388c911ba2021f0ca1bc4287f2ae5ad73372f7"}
Oct 31 19:15:04.816526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.816492    5551 generic.go:356] "Generic (PLEG): container finished" podID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerID="b765254457cb58264d4c3ae070b98612a71b69c31cbf7728850e224bec32419e" exitCode=60
Oct 31 19:15:04.816998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.816567    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" event={"ID":"1819ada6-e365-4e02-9f20-79c8931eb80f","Type":"ContainerDied","Data":"b765254457cb58264d4c3ae070b98612a71b69c31cbf7728850e224bec32419e"}
Oct 31 19:15:04.816998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.816627    5551 scope.go:122] "RemoveContainer" containerID="7481f88194b94520d70bd11f0f7052adf550aef56c26e2e166806bb1e08d85b8"
Oct 31 19:15:04.816998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.816913    5551 scope.go:122] "RemoveContainer" containerID="b765254457cb58264d4c3ae070b98612a71b69c31cbf7728850e224bec32419e"
Oct 31 19:15:04.817222 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:04.817187    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk_odh-ai-gateway-infra(1819ada6-e365-4e02-9f20-79c8931eb80f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f"
Oct 31 19:15:04.819055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.819032    5551 generic.go:356] "Generic (PLEG): container finished" podID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerID="983a2a8fefca644e98f93aaf89f2fed0d50fc8a7bfbdf83792a8585ca6539ead" exitCode=60
Oct 31 19:15:04.819209 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.819120    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" event={"ID":"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6","Type":"ContainerDied","Data":"983a2a8fefca644e98f93aaf89f2fed0d50fc8a7bfbdf83792a8585ca6539ead"}
Oct 31 19:15:04.819382 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.819363    5551 scope.go:122] "RemoveContainer" containerID="983a2a8fefca644e98f93aaf89f2fed0d50fc8a7bfbdf83792a8585ca6539ead"
Oct 31 19:15:04.819731 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:04.819690    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs_odh-ai-gateway-infra(e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6"
Oct 31 19:15:04.821206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.821185    5551 generic.go:356] "Generic (PLEG): container finished" podID="47cde15d-d21e-4c30-b17e-958e7e235910" containerID="35e1bef7737d63ce6ce88a698952abf4ee59cf88a6473faebdcd0f45f40cfbe8" exitCode=60
Oct 31 19:15:04.821310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.821253    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" event={"ID":"47cde15d-d21e-4c30-b17e-958e7e235910","Type":"ContainerDied","Data":"35e1bef7737d63ce6ce88a698952abf4ee59cf88a6473faebdcd0f45f40cfbe8"}
Oct 31 19:15:04.821526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.821511    5551 scope.go:122] "RemoveContainer" containerID="35e1bef7737d63ce6ce88a698952abf4ee59cf88a6473faebdcd0f45f40cfbe8"
Oct 31 19:15:04.821767 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:04.821740    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb_odh-ai-gateway-infra(47cde15d-d21e-4c30-b17e-958e7e235910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" podUID="47cde15d-d21e-4c30-b17e-958e7e235910"
Oct 31 19:15:04.823535 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.823513    5551 generic.go:356] "Generic (PLEG): container finished" podID="b0cde556-93c8-440d-af71-4401046cc720" containerID="b93abc7425c42ff46d82e3f2e8388c911ba2021f0ca1bc4287f2ae5ad73372f7" exitCode=60
Oct 31 19:15:04.823637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.823591    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" event={"ID":"b0cde556-93c8-440d-af71-4401046cc720","Type":"ContainerDied","Data":"b93abc7425c42ff46d82e3f2e8388c911ba2021f0ca1bc4287f2ae5ad73372f7"}
Oct 31 19:15:04.823821 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.823802    5551 scope.go:122] "RemoveContainer" containerID="b93abc7425c42ff46d82e3f2e8388c911ba2021f0ca1bc4287f2ae5ad73372f7"
Oct 31 19:15:04.824060 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:04.824039    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87_odh-ai-gateway-infra(b0cde556-93c8-440d-af71-4401046cc720)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" podUID="b0cde556-93c8-440d-af71-4401046cc720"
Oct 31 19:15:04.825771 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.825749    5551 generic.go:356] "Generic (PLEG): container finished" podID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerID="baa76c6eb5893092d25c2345cd29b3f533af88def947c30fcc66a6d313e90553" exitCode=60
Oct 31 19:15:04.825874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.825837    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" event={"ID":"56756066-3329-4ea4-8759-8dbe2f99fe16","Type":"ContainerDied","Data":"baa76c6eb5893092d25c2345cd29b3f533af88def947c30fcc66a6d313e90553"}
Oct 31 19:15:04.826064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.826045    5551 scope.go:122] "RemoveContainer" containerID="baa76c6eb5893092d25c2345cd29b3f533af88def947c30fcc66a6d313e90553"
Oct 31 19:15:04.826339 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:04.826317    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7_odh-ai-gateway-infra(56756066-3329-4ea4-8759-8dbe2f99fe16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16"
Oct 31 19:15:04.830699 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.830682    5551 scope.go:122] "RemoveContainer" containerID="9820c029ecd4913ddcdff41ae1e8ebada8ff8b918e88c8f5688a2b564afe7104"
Oct 31 19:15:04.846270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.846252    5551 scope.go:122] "RemoveContainer" containerID="7b61173fdbfa5b90ee9e95ce7adf235167ec89378a0883596a2c3db4267c6793"
Oct 31 19:15:04.860118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.860097    5551 scope.go:122] "RemoveContainer" containerID="eef0ac8f989d14931ab03b636b91d540b7077dcd7740b370119ed7bbf739a9d3"
Oct 31 19:15:04.873312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.873290    5551 scope.go:122] "RemoveContainer" containerID="34e3a53502ba5e47dc45aeb27cdfdcee4834c78bb8e91c39eb699533f616a663"
Oct 31 19:15:04.911092 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:04.911062    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" path="/var/lib/kubelet/pods/d0bff25e-b2ff-42a7-bcff-c559781d1d9d/volumes"
Oct 31 19:15:05.836871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:05.836842    5551 scope.go:122] "RemoveContainer" containerID="baa76c6eb5893092d25c2345cd29b3f533af88def947c30fcc66a6d313e90553"
Oct 31 19:15:05.837401 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:05.837202    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7_odh-ai-gateway-infra(56756066-3329-4ea4-8759-8dbe2f99fe16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16"
Oct 31 19:15:05.838508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:05.838452    5551 scope.go:122] "RemoveContainer" containerID="b765254457cb58264d4c3ae070b98612a71b69c31cbf7728850e224bec32419e"
Oct 31 19:15:05.838740 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:05.838719    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk_odh-ai-gateway-infra(1819ada6-e365-4e02-9f20-79c8931eb80f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f"
Oct 31 19:15:05.840126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:05.840103    5551 scope.go:122] "RemoveContainer" containerID="983a2a8fefca644e98f93aaf89f2fed0d50fc8a7bfbdf83792a8585ca6539ead"
Oct 31 19:15:05.840356 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:05.840335    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs_odh-ai-gateway-infra(e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6"
Oct 31 19:15:05.841707 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:05.841688    5551 scope.go:122] "RemoveContainer" containerID="35e1bef7737d63ce6ce88a698952abf4ee59cf88a6473faebdcd0f45f40cfbe8"
Oct 31 19:15:05.841912 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:05.841892    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb_odh-ai-gateway-infra(47cde15d-d21e-4c30-b17e-958e7e235910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" podUID="47cde15d-d21e-4c30-b17e-958e7e235910"
Oct 31 19:15:05.843255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:05.843238    5551 scope.go:122] "RemoveContainer" containerID="b93abc7425c42ff46d82e3f2e8388c911ba2021f0ca1bc4287f2ae5ad73372f7"
Oct 31 19:15:05.843441 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:05.843423    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87_odh-ai-gateway-infra(b0cde556-93c8-440d-af71-4401046cc720)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" podUID="b0cde556-93c8-440d-af71-4401046cc720"
Oct 31 19:15:14.902118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:14.902082    5551 scope.go:122] "RemoveContainer" containerID="b765254457cb58264d4c3ae070b98612a71b69c31cbf7728850e224bec32419e"
Oct 31 19:15:14.902540 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:14.902293    5551 scope.go:122] "RemoveContainer" containerID="b93abc7425c42ff46d82e3f2e8388c911ba2021f0ca1bc4287f2ae5ad73372f7"
Oct 31 19:15:15.901725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:15.901695    5551 scope.go:122] "RemoveContainer" containerID="35e1bef7737d63ce6ce88a698952abf4ee59cf88a6473faebdcd0f45f40cfbe8"
Oct 31 19:15:15.901907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:15.901834    5551 scope.go:122] "RemoveContainer" containerID="983a2a8fefca644e98f93aaf89f2fed0d50fc8a7bfbdf83792a8585ca6539ead"
Oct 31 19:15:15.931097 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:15.931070    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" event={"ID":"b0cde556-93c8-440d-af71-4401046cc720","Type":"ContainerStarted","Data":"ff2e9a3f19741cf93e641af787950cf2b0ade5fc131197ab347ea843e4659c73"}
Oct 31 19:15:15.933036 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:15.933014    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" event={"ID":"1819ada6-e365-4e02-9f20-79c8931eb80f","Type":"ContainerStarted","Data":"8817e043921db81b3274dc95bbee0ea9508ee8faad29fd95d9ed9ab34c136b96"}
Oct 31 19:15:16.605228 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:16.605186    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:15:16.906086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.906057    5551 scope.go:122] "RemoveContainer" containerID="baa76c6eb5893092d25c2345cd29b3f533af88def947c30fcc66a6d313e90553"
Oct 31 19:15:16.941697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.941669    5551 generic.go:356] "Generic (PLEG): container finished" podID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerID="8817e043921db81b3274dc95bbee0ea9508ee8faad29fd95d9ed9ab34c136b96" exitCode=60
Oct 31 19:15:16.942034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.941743    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" event={"ID":"1819ada6-e365-4e02-9f20-79c8931eb80f","Type":"ContainerDied","Data":"8817e043921db81b3274dc95bbee0ea9508ee8faad29fd95d9ed9ab34c136b96"}
Oct 31 19:15:16.942034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.941812    5551 scope.go:122] "RemoveContainer" containerID="b765254457cb58264d4c3ae070b98612a71b69c31cbf7728850e224bec32419e"
Oct 31 19:15:16.942188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.942172    5551 scope.go:122] "RemoveContainer" containerID="8817e043921db81b3274dc95bbee0ea9508ee8faad29fd95d9ed9ab34c136b96"
Oct 31 19:15:16.942489 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:16.942449    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk_odh-ai-gateway-infra(1819ada6-e365-4e02-9f20-79c8931eb80f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f"
Oct 31 19:15:16.943737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.943716    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" event={"ID":"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6","Type":"ContainerStarted","Data":"efd6d5ec6c06cc0b2ed4ecb61b48e36c4ceef30e9723554e1bbc66513ad74d18"}
Oct 31 19:15:16.945693 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.945667    5551 generic.go:356] "Generic (PLEG): container finished" podID="47cde15d-d21e-4c30-b17e-958e7e235910" containerID="129fbd68b3360f3e02b2d57d66cc4afdc66c70bddfa546d284a7639489e82897" exitCode=60
Oct 31 19:15:16.945808 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.945738    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" event={"ID":"47cde15d-d21e-4c30-b17e-958e7e235910","Type":"ContainerDied","Data":"129fbd68b3360f3e02b2d57d66cc4afdc66c70bddfa546d284a7639489e82897"}
Oct 31 19:15:16.946069 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.946053    5551 scope.go:122] "RemoveContainer" containerID="129fbd68b3360f3e02b2d57d66cc4afdc66c70bddfa546d284a7639489e82897"
Oct 31 19:15:16.946316 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:16.946287    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb_odh-ai-gateway-infra(47cde15d-d21e-4c30-b17e-958e7e235910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" podUID="47cde15d-d21e-4c30-b17e-958e7e235910"
Oct 31 19:15:16.948598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.948573    5551 generic.go:356] "Generic (PLEG): container finished" podID="b0cde556-93c8-440d-af71-4401046cc720" containerID="ff2e9a3f19741cf93e641af787950cf2b0ade5fc131197ab347ea843e4659c73" exitCode=60
Oct 31 19:15:16.948709 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.948615    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" event={"ID":"b0cde556-93c8-440d-af71-4401046cc720","Type":"ContainerDied","Data":"ff2e9a3f19741cf93e641af787950cf2b0ade5fc131197ab347ea843e4659c73"}
Oct 31 19:15:16.948953 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.948930    5551 scope.go:122] "RemoveContainer" containerID="ff2e9a3f19741cf93e641af787950cf2b0ade5fc131197ab347ea843e4659c73"
Oct 31 19:15:16.949215 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:16.949193    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87_odh-ai-gateway-infra(b0cde556-93c8-440d-af71-4401046cc720)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" podUID="b0cde556-93c8-440d-af71-4401046cc720"
Oct 31 19:15:16.956979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.956960    5551 scope.go:122] "RemoveContainer" containerID="35e1bef7737d63ce6ce88a698952abf4ee59cf88a6473faebdcd0f45f40cfbe8"
Oct 31 19:15:16.969628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:16.969612    5551 scope.go:122] "RemoveContainer" containerID="b93abc7425c42ff46d82e3f2e8388c911ba2021f0ca1bc4287f2ae5ad73372f7"
Oct 31 19:15:17.063554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.063518    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"]
Oct 31 19:15:17.176048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.176015    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"]
Oct 31 19:15:17.959977 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.959945    5551 generic.go:356] "Generic (PLEG): container finished" podID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerID="efd6d5ec6c06cc0b2ed4ecb61b48e36c4ceef30e9723554e1bbc66513ad74d18" exitCode=60
Oct 31 19:15:17.960510 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.960025    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" event={"ID":"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6","Type":"ContainerDied","Data":"efd6d5ec6c06cc0b2ed4ecb61b48e36c4ceef30e9723554e1bbc66513ad74d18"}
Oct 31 19:15:17.960510 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.960091    5551 scope.go:122] "RemoveContainer" containerID="983a2a8fefca644e98f93aaf89f2fed0d50fc8a7bfbdf83792a8585ca6539ead"
Oct 31 19:15:17.960510 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.960481    5551 scope.go:122] "RemoveContainer" containerID="efd6d5ec6c06cc0b2ed4ecb61b48e36c4ceef30e9723554e1bbc66513ad74d18"
Oct 31 19:15:17.960867 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:17.960837    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs_odh-ai-gateway-infra(e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6"
Oct 31 19:15:17.965882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.965856    5551 generic.go:356] "Generic (PLEG): container finished" podID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerID="66c0688353083985cdbfd7bbc405f88bc0c9a59a0109d1d9743f8c69d24dfe7c" exitCode=60
Oct 31 19:15:17.966001 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.965893    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" event={"ID":"56756066-3329-4ea4-8759-8dbe2f99fe16","Type":"ContainerDied","Data":"66c0688353083985cdbfd7bbc405f88bc0c9a59a0109d1d9743f8c69d24dfe7c"}
Oct 31 19:15:17.966554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.966532    5551 scope.go:122] "RemoveContainer" containerID="66c0688353083985cdbfd7bbc405f88bc0c9a59a0109d1d9743f8c69d24dfe7c"
Oct 31 19:15:17.966964 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:17.966933    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7_odh-ai-gateway-infra(56756066-3329-4ea4-8759-8dbe2f99fe16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16"
Oct 31 19:15:17.976710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:17.976650    5551 scope.go:122] "RemoveContainer" containerID="baa76c6eb5893092d25c2345cd29b3f533af88def947c30fcc66a6d313e90553"
Oct 31 19:15:18.135556 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.135532    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:18.179935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.179910    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"]
Oct 31 19:15:18.182380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.182364    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:18.195782 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.195762    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1819ada6-e365-4e02-9f20-79c8931eb80f-kube-api-access-cc8qn\" (UniqueName: \"kubernetes.io/projected/1819ada6-e365-4e02-9f20-79c8931eb80f-kube-api-access-cc8qn\") pod \"1819ada6-e365-4e02-9f20-79c8931eb80f\" (UID: \"1819ada6-e365-4e02-9f20-79c8931eb80f\") "
Oct 31 19:15:18.195892 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.195862    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1819ada6-e365-4e02-9f20-79c8931eb80f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1819ada6-e365-4e02-9f20-79c8931eb80f-maas-api-service-ca\") pod \"1819ada6-e365-4e02-9f20-79c8931eb80f\" (UID: \"1819ada6-e365-4e02-9f20-79c8931eb80f\") "
Oct 31 19:15:18.196178 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.196149    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1819ada6-e365-4e02-9f20-79c8931eb80f-maas-api-service-ca" pod "1819ada6-e365-4e02-9f20-79c8931eb80f" (UID: "1819ada6-e365-4e02-9f20-79c8931eb80f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:18.196306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.196293    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1819ada6-e365-4e02-9f20-79c8931eb80f-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.198024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.198004    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1819ada6-e365-4e02-9f20-79c8931eb80f-kube-api-access-cc8qn" pod "1819ada6-e365-4e02-9f20-79c8931eb80f" (UID: "1819ada6-e365-4e02-9f20-79c8931eb80f"). InnerVolumeSpecName "kube-api-access-cc8qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:18.293256 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.293229    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"]
Oct 31 19:15:18.297702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.297671    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cde556-93c8-440d-af71-4401046cc720-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cde556-93c8-440d-af71-4401046cc720-maas-api-service-ca\") pod \"b0cde556-93c8-440d-af71-4401046cc720\" (UID: \"b0cde556-93c8-440d-af71-4401046cc720\") "
Oct 31 19:15:18.297934 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.297736    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cde556-93c8-440d-af71-4401046cc720-kube-api-access-lpjl4\" (UniqueName: \"kubernetes.io/projected/b0cde556-93c8-440d-af71-4401046cc720-kube-api-access-lpjl4\") pod \"b0cde556-93c8-440d-af71-4401046cc720\" (UID: \"b0cde556-93c8-440d-af71-4401046cc720\") "
Oct 31 19:15:18.298263 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.298179    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cde556-93c8-440d-af71-4401046cc720-maas-api-service-ca" pod "b0cde556-93c8-440d-af71-4401046cc720" (UID: "b0cde556-93c8-440d-af71-4401046cc720"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:18.298806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.298782    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc8qn\" (UniqueName: \"kubernetes.io/projected/1819ada6-e365-4e02-9f20-79c8931eb80f-kube-api-access-cc8qn\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.298898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.298811    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cde556-93c8-440d-af71-4401046cc720-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.300145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.300123    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cde556-93c8-440d-af71-4401046cc720-kube-api-access-lpjl4" pod "b0cde556-93c8-440d-af71-4401046cc720" (UID: "b0cde556-93c8-440d-af71-4401046cc720"). InnerVolumeSpecName "kube-api-access-lpjl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:18.399971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.399925    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjl4\" (UniqueName: \"kubernetes.io/projected/b0cde556-93c8-440d-af71-4401046cc720-kube-api-access-lpjl4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.433553 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.433523    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:18.501335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.501202    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cde15d-d21e-4c30-b17e-958e7e235910-kube-api-access-kqz6d\" (UniqueName: \"kubernetes.io/projected/47cde15d-d21e-4c30-b17e-958e7e235910-kube-api-access-kqz6d\") pod \"47cde15d-d21e-4c30-b17e-958e7e235910\" (UID: \"47cde15d-d21e-4c30-b17e-958e7e235910\") "
Oct 31 19:15:18.501652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.501593    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cde15d-d21e-4c30-b17e-958e7e235910-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cde15d-d21e-4c30-b17e-958e7e235910-maas-api-service-ca\") pod \"47cde15d-d21e-4c30-b17e-958e7e235910\" (UID: \"47cde15d-d21e-4c30-b17e-958e7e235910\") "
Oct 31 19:15:18.502014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.501985    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cde15d-d21e-4c30-b17e-958e7e235910-maas-api-service-ca" pod "47cde15d-d21e-4c30-b17e-958e7e235910" (UID: "47cde15d-d21e-4c30-b17e-958e7e235910"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:18.502352 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.502332    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cde15d-d21e-4c30-b17e-958e7e235910-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.504070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.504033    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cde15d-d21e-4c30-b17e-958e7e235910-kube-api-access-kqz6d" pod "47cde15d-d21e-4c30-b17e-958e7e235910" (UID: "47cde15d-d21e-4c30-b17e-958e7e235910"). InnerVolumeSpecName "kube-api-access-kqz6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:18.603766 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.603734    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqz6d\" (UniqueName: \"kubernetes.io/projected/47cde15d-d21e-4c30-b17e-958e7e235910-kube-api-access-kqz6d\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.976386 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.976360    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"
Oct 31 19:15:18.976827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.976315    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb" event={"ID":"47cde15d-d21e-4c30-b17e-958e7e235910","Type":"ContainerDied","Data":"a385e5b5add16ee940d7bdc255477f4d3cead63d84f8965c5a418ca156e579f2"}
Oct 31 19:15:18.976827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.976564    5551 scope.go:122] "RemoveContainer" containerID="129fbd68b3360f3e02b2d57d66cc4afdc66c70bddfa546d284a7639489e82897"
Oct 31 19:15:18.979190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.978630    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"
Oct 31 19:15:18.979190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.978775    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87" event={"ID":"b0cde556-93c8-440d-af71-4401046cc720","Type":"ContainerDied","Data":"c4ba5032a1e8e19a3bb0ae3770d2936be89fc7e64413e781577a63d9db51bdf1"}
Oct 31 19:15:18.982227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.982206    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"
Oct 31 19:15:18.982347 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.982235    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk" event={"ID":"1819ada6-e365-4e02-9f20-79c8931eb80f","Type":"ContainerDied","Data":"8523a7072d5babc10e2bcbba9c0374cac1e069be184b5144d080ae6132ee7cb8"}
Oct 31 19:15:18.993494 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:18.991419    5551 scope.go:122] "RemoveContainer" containerID="ff2e9a3f19741cf93e641af787950cf2b0ade5fc131197ab347ea843e4659c73"
Oct 31 19:15:19.002404 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.002385    5551 scope.go:122] "RemoveContainer" containerID="8817e043921db81b3274dc95bbee0ea9508ee8faad29fd95d9ed9ab34c136b96"
Oct 31 19:15:19.138078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.138046    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:19.141034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.141007    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"]
Oct 31 19:15:19.146121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.146091    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891235-t6lkb"]
Oct 31 19:15:19.211215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.211144    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-kube-api-access-dsbfc\" (UniqueName: \"kubernetes.io/projected/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-kube-api-access-dsbfc\") pod \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\" (UID: \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\") "
Oct 31 19:15:19.211401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.211218    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-maas-api-service-ca\") pod \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\" (UID: \"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6\") "
Oct 31 19:15:19.211676 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.211649    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-maas-api-service-ca" pod "e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" (UID: "e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:19.211816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.211798    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:19.213864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.213834    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-kube-api-access-dsbfc" pod "e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" (UID: "e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6"). InnerVolumeSpecName "kube-api-access-dsbfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:19.253439 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.253381    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"]
Oct 31 19:15:19.256990 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.256963    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891235-82b87"]
Oct 31 19:15:19.313423 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.313372    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsbfc\" (UniqueName: \"kubernetes.io/projected/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6-kube-api-access-dsbfc\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:19.366736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.366705    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"]
Oct 31 19:15:19.370036 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.370008    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891235-dn5tk"]
Oct 31 19:15:19.426625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.426589    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"]
Oct 31 19:15:19.605008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.604969    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:19.718527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.718496    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56756066-3329-4ea4-8759-8dbe2f99fe16-kube-api-access-m4tz8\" (UniqueName: \"kubernetes.io/projected/56756066-3329-4ea4-8759-8dbe2f99fe16-kube-api-access-m4tz8\") pod \"56756066-3329-4ea4-8759-8dbe2f99fe16\" (UID: \"56756066-3329-4ea4-8759-8dbe2f99fe16\") "
Oct 31 19:15:19.718719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.718598    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56756066-3329-4ea4-8759-8dbe2f99fe16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56756066-3329-4ea4-8759-8dbe2f99fe16-maas-api-service-ca\") pod \"56756066-3329-4ea4-8759-8dbe2f99fe16\" (UID: \"56756066-3329-4ea4-8759-8dbe2f99fe16\") "
Oct 31 19:15:19.719034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.719001    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56756066-3329-4ea4-8759-8dbe2f99fe16-maas-api-service-ca" pod "56756066-3329-4ea4-8759-8dbe2f99fe16" (UID: "56756066-3329-4ea4-8759-8dbe2f99fe16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:19.720808 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.720783    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56756066-3329-4ea4-8759-8dbe2f99fe16-kube-api-access-m4tz8" pod "56756066-3329-4ea4-8759-8dbe2f99fe16" (UID: "56756066-3329-4ea4-8759-8dbe2f99fe16"). InnerVolumeSpecName "kube-api-access-m4tz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:19.820718 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.820634    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4tz8\" (UniqueName: \"kubernetes.io/projected/56756066-3329-4ea4-8759-8dbe2f99fe16-kube-api-access-m4tz8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:19.820718 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:19.820670    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56756066-3329-4ea4-8759-8dbe2f99fe16-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:20.012356 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.012318    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs" event={"ID":"e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6","Type":"ContainerDied","Data":"5df0838150ec8817d25ea35dcfc6268f1b02ee9e598dad6589a55b893758ef99"}
Oct 31 19:15:20.013018 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.012365    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"
Oct 31 19:15:20.013155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.013137    5551 scope.go:122] "RemoveContainer" containerID="efd6d5ec6c06cc0b2ed4ecb61b48e36c4ceef30e9723554e1bbc66513ad74d18"
Oct 31 19:15:20.026725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.024338    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7" event={"ID":"56756066-3329-4ea4-8759-8dbe2f99fe16","Type":"ContainerDied","Data":"845a7f27813dac3dc28584c9086bf5ef74d6bf786adda1c7b8d07545b32823ba"}
Oct 31 19:15:20.026725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.024369    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"
Oct 31 19:15:20.038337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.038310    5551 scope.go:122] "RemoveContainer" containerID="66c0688353083985cdbfd7bbc405f88bc0c9a59a0109d1d9743f8c69d24dfe7c"
Oct 31 19:15:20.178296 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.178259    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"]
Oct 31 19:15:20.181979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.181954    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6cfd68-29891235-pgjqs"]
Oct 31 19:15:20.290363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.290331    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"]
Oct 31 19:15:20.293466 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.293427    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891235-hhxl7"]
Oct 31 19:15:20.910542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.910508    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" path="/var/lib/kubelet/pods/1819ada6-e365-4e02-9f20-79c8931eb80f/volumes"
Oct 31 19:15:20.911030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.911015    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" path="/var/lib/kubelet/pods/47cde15d-d21e-4c30-b17e-958e7e235910/volumes"
Oct 31 19:15:20.911389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.911377    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" path="/var/lib/kubelet/pods/56756066-3329-4ea4-8759-8dbe2f99fe16/volumes"
Oct 31 19:15:20.912120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.912104    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cde556-93c8-440d-af71-4401046cc720" path="/var/lib/kubelet/pods/b0cde556-93c8-440d-af71-4401046cc720/volumes"
Oct 31 19:15:20.912528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:20.912509    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" path="/var/lib/kubelet/pods/e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6/volumes"
Oct 31 19:15:30.004686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:30.004647    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"]
Oct 31 19:15:45.030081 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.029973    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5" podUID="a7924fe1-e5b6-45ae-8814-ec2b596f5954" containerName="main" containerID="cri-o://8ccabdad42c3c9a8ff8b4e4855bbd209a1f42e83c41502c7f645885fd3a12bb1" gracePeriod=45
Oct 31 19:15:45.281923 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.281827    5551 generic.go:356] "Generic (PLEG): container finished" podID="a7924fe1-e5b6-45ae-8814-ec2b596f5954" containerID="8ccabdad42c3c9a8ff8b4e4855bbd209a1f42e83c41502c7f645885fd3a12bb1" exitCode=0
Oct 31 19:15:45.282283 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.282258    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5" event={"ID":"a7924fe1-e5b6-45ae-8814-ec2b596f5954","Type":"ContainerDied","Data":"8ccabdad42c3c9a8ff8b4e4855bbd209a1f42e83c41502c7f645885fd3a12bb1"}
Oct 31 19:15:45.317594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.317565    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:15:45.407551 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.407505    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-model-cache\") pod \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") "
Oct 31 19:15:45.407767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.407668    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-home\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-home\") pod \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") "
Oct 31 19:15:45.407767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.407733    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-dshm\") pod \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") "
Oct 31 19:15:45.407907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.407766    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-model-cache" pod "a7924fe1-e5b6-45ae-8814-ec2b596f5954" (UID: "a7924fe1-e5b6-45ae-8814-ec2b596f5954"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:45.407907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.407801    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7924fe1-e5b6-45ae-8814-ec2b596f5954-kube-api-access-cpm64\" (UniqueName: \"kubernetes.io/projected/a7924fe1-e5b6-45ae-8814-ec2b596f5954-kube-api-access-cpm64\") pod \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") "
Oct 31 19:15:45.407907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.407855    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tmp-dir\") pod \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") "
Oct 31 19:15:45.407907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.407896    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tls-certs\") pod \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\" (UID: \"a7924fe1-e5b6-45ae-8814-ec2b596f5954\") "
Oct 31 19:15:45.408124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.407942    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-home" pod "a7924fe1-e5b6-45ae-8814-ec2b596f5954" (UID: "a7924fe1-e5b6-45ae-8814-ec2b596f5954"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:45.408124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.408102    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tmp-dir" pod "a7924fe1-e5b6-45ae-8814-ec2b596f5954" (UID: "a7924fe1-e5b6-45ae-8814-ec2b596f5954"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:45.408579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.408476    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:45.408579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.408511    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:45.408579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.408526    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:45.410395 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.410371    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tls-certs" pod "a7924fe1-e5b6-45ae-8814-ec2b596f5954" (UID: "a7924fe1-e5b6-45ae-8814-ec2b596f5954"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:15:45.410557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.410542    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-dshm" pod "a7924fe1-e5b6-45ae-8814-ec2b596f5954" (UID: "a7924fe1-e5b6-45ae-8814-ec2b596f5954"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:15:45.410610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.410569    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7924fe1-e5b6-45ae-8814-ec2b596f5954-kube-api-access-cpm64" pod "a7924fe1-e5b6-45ae-8814-ec2b596f5954" (UID: "a7924fe1-e5b6-45ae-8814-ec2b596f5954"). InnerVolumeSpecName "kube-api-access-cpm64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:45.509631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.509593    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7924fe1-e5b6-45ae-8814-ec2b596f5954-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:45.509631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.509623    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpm64\" (UniqueName: \"kubernetes.io/projected/a7924fe1-e5b6-45ae-8814-ec2b596f5954-kube-api-access-cpm64\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:45.509631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:45.509634    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7924fe1-e5b6-45ae-8814-ec2b596f5954-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:46.290291 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:46.290262    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"
Oct 31 19:15:46.290291 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:46.290281    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5" event={"ID":"a7924fe1-e5b6-45ae-8814-ec2b596f5954","Type":"ContainerDied","Data":"a82116b177e51d8b32047b5c189e9a4a93628287937cbac7394bd2857786e049"}
Oct 31 19:15:46.290815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:46.290337    5551 scope.go:122] "RemoveContainer" containerID="8ccabdad42c3c9a8ff8b4e4855bbd209a1f42e83c41502c7f645885fd3a12bb1"
Oct 31 19:15:46.454559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:46.454525    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"]
Oct 31 19:15:46.501315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:46.501283    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d8f34e/auth-test-model-d8f34e-kserve-d64bc64c7-pvhm5"]
Oct 31 19:15:46.604679 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:46.604581    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:15:46.910647 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:46.910615    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7924fe1-e5b6-45ae-8814-ec2b596f5954" path="/var/lib/kubelet/pods/a7924fe1-e5b6-45ae-8814-ec2b596f5954/volumes"
Oct 31 19:15:49.754157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754125    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7554-122db0"]
Oct 31 19:15:49.754739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754724    5551 cpu_manager.go:414] "containerMap: removing container" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:49.754786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754740    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:49.754786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754748    5551 cpu_manager.go:414] "containerMap: removing container" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:49.754786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754753    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:49.754786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754770    5551 cpu_manager.go:414] "containerMap: removing container" podUID="a7924fe1-e5b6-45ae-8814-ec2b596f5954" containerName="main"
Oct 31 19:15:49.754786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754775    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7924fe1-e5b6-45ae-8814-ec2b596f5954" containerName="main"
Oct 31 19:15:49.754786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754781    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:49.754786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754785    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754793    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754798    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754813    5551 cpu_manager.go:414] "containerMap: removing container" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754818    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754831    5551 cpu_manager.go:414] "containerMap: removing container" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754835    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754842    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754846    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754854    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754859    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754871    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754876    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754882    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754887    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754896    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754900    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754910    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754916    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754928    5551 cpu_manager.go:414] "containerMap: removing container" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:49.755005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.754933    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755035    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755048    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755057    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755067    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755075    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755082    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755087    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755093    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755098    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755105    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755111    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7924fe1-e5b6-45ae-8814-ec2b596f5954" containerName="main"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755117    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bff25e-b2ff-42a7-bcff-c559781d1d9d" containerName="istio-proxy"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755125    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:49.755626 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.755131    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:49.759848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.759825    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:49.814845 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.814815    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-122db0"]
Oct 31 19:15:49.860098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.860066    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bg2\""
Oct 31 19:15:49.861800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.861775    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4qq\" (UniqueName: \"kubernetes.io/projected/83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42-kube-api-access-kv4qq\") pod \"test-curl-7554-122db0\" (UID: \"83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42\") " pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:49.963409 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:49.963365    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4qq\" (UniqueName: \"kubernetes.io/projected/83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42-kube-api-access-kv4qq\") pod \"test-curl-7554-122db0\" (UID: \"83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42\") " pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:50.022048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:50.021980    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4qq\" (UniqueName: \"kubernetes.io/projected/83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42-kube-api-access-kv4qq\") pod \"test-curl-7554-122db0\" (UID: \"83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42\") " pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:50.075773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:50.075726    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:50.513797 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:50.513771    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-122db0"]
Oct 31 19:15:50.515395 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:15:50.515371    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83bbabbb_ae2d_4b29_98b5_dc0fae4c1d42.slice/crio-6e90b398e73ead943075c22481328e76be72c7d956d342522e67a67c47538053 WatchSource:0}: Error finding container 6e90b398e73ead943075c22481328e76be72c7d956d342522e67a67c47538053: Status 404 returned error can't find the container with id 6e90b398e73ead943075c22481328e76be72c7d956d342522e67a67c47538053
Oct 31 19:15:50.517281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:50.517264    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:15:50.913810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:50.913779    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"]
Oct 31 19:15:51.339188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:51.339102    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-122db0" event={"ID":"83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42","Type":"ContainerStarted","Data":"6e90b398e73ead943075c22481328e76be72c7d956d342522e67a67c47538053"}
Oct 31 19:15:54.365473 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:54.365416    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-122db0" event={"ID":"83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42","Type":"ContainerStarted","Data":"cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8"}
Oct 31 19:15:54.478132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:54.478077    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7554-122db0" podStartSLOduration=2.564672013 podStartE2EDuration="5.478062027s" totalImagesPullingTime="2.913390014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:54.477227856 +0000 UTC m=+3257.708466146" watchObservedRunningTime="2026-10-31 19:15:54.478062027 +0000 UTC m=+3257.709300317"
Oct 31 19:15:56.045067 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.045032    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7554-122db0"]
Oct 31 19:15:56.051260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.051224    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7554-122db0"]
Oct 31 19:15:56.380316 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.380274    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7554-122db0" podUID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" containerName="test-curl-7554-122db0" containerID="cri-o://cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8" gracePeriod=2
Oct 31 19:15:56.431927 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:56.431888    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-122db0\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:56.449373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.449346    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7554-7c5beb"]
Oct 31 19:15:56.450160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450140    5551 cpu_manager.go:414] "containerMap: removing container" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:56.450253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450163    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cde15d-d21e-4c30-b17e-958e7e235910" containerName="cleanup"
Oct 31 19:15:56.450253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450184    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:56.450253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450192    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:56.450253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450248    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:56.450436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450256    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1819ada6-e365-4e02-9f20-79c8931eb80f" containerName="cleanup"
Oct 31 19:15:56.450436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450268    5551 cpu_manager.go:414] "containerMap: removing container" podUID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" containerName="test-curl-7554-122db0"
Oct 31 19:15:56.450436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450276    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" containerName="test-curl-7554-122db0"
Oct 31 19:15:56.450436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450412    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44ac2e8-39d2-4260-b6ea-9fd9f9fceac6" containerName="cleanup"
Oct 31 19:15:56.450436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450423    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" containerName="test-curl-7554-122db0"
Oct 31 19:15:56.450436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450434    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="56756066-3329-4ea4-8759-8dbe2f99fe16" containerName="cleanup"
Oct 31 19:15:56.450766 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.450446    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cde556-93c8-440d-af71-4401046cc720" containerName="cleanup"
Oct 31 19:15:56.454554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.454534    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-7c5beb"
Oct 31 19:15:56.505428 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:56.505389    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-122db0\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:56.509579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.509558    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-7c5beb"]
Oct 31 19:15:56.552096 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.552057    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8rn\" (UniqueName: \"kubernetes.io/projected/5804c8d0-5ea5-45b8-b30b-6b723cff5a9f-kube-api-access-bg8rn\") pod \"test-curl-7554-7c5beb\" (UID: \"5804c8d0-5ea5-45b8-b30b-6b723cff5a9f\") " pod="opendatahub/test-curl-7554-7c5beb"
Oct 31 19:15:56.653744 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.653639    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8rn\" (UniqueName: \"kubernetes.io/projected/5804c8d0-5ea5-45b8-b30b-6b723cff5a9f-kube-api-access-bg8rn\") pod \"test-curl-7554-7c5beb\" (UID: \"5804c8d0-5ea5-45b8-b30b-6b723cff5a9f\") " pod="opendatahub/test-curl-7554-7c5beb"
Oct 31 19:15:56.711534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.711507    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8rn\" (UniqueName: \"kubernetes.io/projected/5804c8d0-5ea5-45b8-b30b-6b723cff5a9f-kube-api-access-bg8rn\") pod \"test-curl-7554-7c5beb\" (UID: \"5804c8d0-5ea5-45b8-b30b-6b723cff5a9f\") " pod="opendatahub/test-curl-7554-7c5beb"
Oct 31 19:15:56.766973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:56.766940    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-7c5beb"
Oct 31 19:15:56.958290 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:56.958195    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-122db0\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:57.143914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:57.143887    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-7c5beb"]
Oct 31 19:15:57.144998 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:15:57.144973    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5804c8d0_5ea5_45b8_b30b_6b723cff5a9f.slice/crio-73953c5552b3bede825995839b2df0790007ee74c2cfa6601f42b8a54ff4bf77 WatchSource:0}: Error finding container 73953c5552b3bede825995839b2df0790007ee74c2cfa6601f42b8a54ff4bf77: Status 404 returned error can't find the container with id 73953c5552b3bede825995839b2df0790007ee74c2cfa6601f42b8a54ff4bf77
Oct 31 19:15:57.389426 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:57.389374    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-7c5beb" event={"ID":"5804c8d0-5ea5-45b8-b30b-6b723cff5a9f","Type":"ContainerStarted","Data":"28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6"}
Oct 31 19:15:57.389620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:57.389438    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-7c5beb" event={"ID":"5804c8d0-5ea5-45b8-b30b-6b723cff5a9f","Type":"ContainerStarted","Data":"73953c5552b3bede825995839b2df0790007ee74c2cfa6601f42b8a54ff4bf77"}
Oct 31 19:15:57.503477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:57.503347    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7554-7c5beb" podStartSLOduration=1.503333333 podStartE2EDuration="1.503333333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:57.501923122 +0000 UTC m=+3260.733161413" watchObservedRunningTime="2026-10-31 19:15:57.503333333 +0000 UTC m=+3260.734571622"
Oct 31 19:15:58.644672 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:58.644602    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:58.781695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:58.781645    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42-kube-api-access-kv4qq\" (UniqueName: \"kubernetes.io/projected/83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42-kube-api-access-kv4qq\") pod \"83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42\" (UID: \"83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42\") "
Oct 31 19:15:58.784043 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:58.784011    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42-kube-api-access-kv4qq" pod "83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" (UID: "83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42"). InnerVolumeSpecName "kube-api-access-kv4qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:58.868723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:58.868536    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7554-7c5beb"]
Oct 31 19:15:58.871695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:58.871675    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7554-7c5beb"]
Oct 31 19:15:58.883488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:58.883432    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4qq\" (UniqueName: \"kubernetes.io/projected/83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42-kube-api-access-kv4qq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:58.911309 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:58.911240    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" path="/var/lib/kubelet/pods/83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42/volumes"
Oct 31 19:15:59.017916 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.017884    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7554-624f63"]
Oct 31 19:15:59.018443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.018430    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5804c8d0-5ea5-45b8-b30b-6b723cff5a9f" containerName="test-curl-7554-7c5beb"
Oct 31 19:15:59.018522 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.018444    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5804c8d0-5ea5-45b8-b30b-6b723cff5a9f" containerName="test-curl-7554-7c5beb"
Oct 31 19:15:59.018606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.018596    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5804c8d0-5ea5-45b8-b30b-6b723cff5a9f" containerName="test-curl-7554-7c5beb"
Oct 31 19:15:59.022059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.022044    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:15:59.078853 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.078820    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-624f63"]
Oct 31 19:15:59.086849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.086817    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q67sz\" (UniqueName: \"kubernetes.io/projected/03ccd8e6-9aba-43d3-811d-68d543aa7d71-kube-api-access-q67sz\") pod \"test-curl-7554-624f63\" (UID: \"03ccd8e6-9aba-43d3-811d-68d543aa7d71\") " pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:15:59.187736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.187656    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67sz\" (UniqueName: \"kubernetes.io/projected/03ccd8e6-9aba-43d3-811d-68d543aa7d71-kube-api-access-q67sz\") pod \"test-curl-7554-624f63\" (UID: \"03ccd8e6-9aba-43d3-811d-68d543aa7d71\") " pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:15:59.245525 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.245498    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67sz\" (UniqueName: \"kubernetes.io/projected/03ccd8e6-9aba-43d3-811d-68d543aa7d71-kube-api-access-q67sz\") pod \"test-curl-7554-624f63\" (UID: \"03ccd8e6-9aba-43d3-811d-68d543aa7d71\") " pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:15:59.334414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.334374    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:15:59.412521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.410632    5551 generic.go:356] "Generic (PLEG): container finished" podID="83bbabbb-ae2d-4b29-98b5-dc0fae4c1d42" containerID="cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8" exitCode=137
Oct 31 19:15:59.412521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.410747    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7554-7c5beb" podUID="5804c8d0-5ea5-45b8-b30b-6b723cff5a9f" containerName="test-curl-7554-7c5beb" containerID="cri-o://28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6" gracePeriod=2
Oct 31 19:15:59.412521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.411053    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-122db0"
Oct 31 19:15:59.412521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.411510    5551 scope.go:122] "RemoveContainer" containerID="cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8"
Oct 31 19:15:59.423651 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.423625    5551 scope.go:122] "RemoveContainer" containerID="cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8"
Oct 31 19:15:59.423920 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:15:59.423901    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8\": container with ID starting with cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8 not found: ID does not exist" containerID="cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8"
Oct 31 19:15:59.423997 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.423928    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8"} err="failed to get container status \"cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8\": rpc error: code = NotFound desc = could not find container \"cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8\": container with ID starting with cd70da959f31b8a83f42fa736011b23f94e789cbda3b729f06651c98e216e0a8 not found: ID does not exist"
Oct 31 19:15:59.720527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:15:59.720494    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-624f63"]
Oct 31 19:15:59.721827 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:15:59.721800    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03ccd8e6_9aba_43d3_811d_68d543aa7d71.slice/crio-99356e65edc9c5038b0ad9fa890aa4193b30983e1f257c0c127147e9e403ac10 WatchSource:0}: Error finding container 99356e65edc9c5038b0ad9fa890aa4193b30983e1f257c0c127147e9e403ac10: Status 404 returned error can't find the container with id 99356e65edc9c5038b0ad9fa890aa4193b30983e1f257c0c127147e9e403ac10
Oct 31 19:16:00.418204 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:00.418164    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-624f63" event={"ID":"03ccd8e6-9aba-43d3-811d-68d543aa7d71","Type":"ContainerStarted","Data":"d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d"}
Oct 31 19:16:00.418384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:00.418217    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-624f63" event={"ID":"03ccd8e6-9aba-43d3-811d-68d543aa7d71","Type":"ContainerStarted","Data":"99356e65edc9c5038b0ad9fa890aa4193b30983e1f257c0c127147e9e403ac10"}
Oct 31 19:16:00.531855 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:00.531806    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7554-624f63" podStartSLOduration=2.531791852 podStartE2EDuration="2.531791852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:00.529312884 +0000 UTC m=+3263.760551176" watchObservedRunningTime="2026-10-31 19:16:00.531791852 +0000 UTC m=+3263.763030141"
Oct 31 19:16:01.651143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:01.651109    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-7c5beb"
Oct 31 19:16:01.721938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:01.721844    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5804c8d0-5ea5-45b8-b30b-6b723cff5a9f-kube-api-access-bg8rn\" (UniqueName: \"kubernetes.io/projected/5804c8d0-5ea5-45b8-b30b-6b723cff5a9f-kube-api-access-bg8rn\") pod \"5804c8d0-5ea5-45b8-b30b-6b723cff5a9f\" (UID: \"5804c8d0-5ea5-45b8-b30b-6b723cff5a9f\") "
Oct 31 19:16:01.724515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:01.724485    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5804c8d0-5ea5-45b8-b30b-6b723cff5a9f-kube-api-access-bg8rn" pod "5804c8d0-5ea5-45b8-b30b-6b723cff5a9f" (UID: "5804c8d0-5ea5-45b8-b30b-6b723cff5a9f"). InnerVolumeSpecName "kube-api-access-bg8rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:01.823526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:01.823488    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg8rn\" (UniqueName: \"kubernetes.io/projected/5804c8d0-5ea5-45b8-b30b-6b723cff5a9f-kube-api-access-bg8rn\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:01.990720 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:01.990369    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7554-624f63"]
Oct 31 19:16:01.994583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:01.994550    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7554-624f63"]
Oct 31 19:16:02.131639 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.131607    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7554-f14a62"]
Oct 31 19:16:02.132194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.132179    5551 cpu_manager.go:414] "containerMap: removing container" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" containerName="test-curl-7554-624f63"
Oct 31 19:16:02.132243 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.132195    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" containerName="test-curl-7554-624f63"
Oct 31 19:16:02.132310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.132299    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" containerName="test-curl-7554-624f63"
Oct 31 19:16:02.135614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.135596    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:02.192338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.192299    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-f14a62"]
Oct 31 19:16:02.229707 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.229674    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9xd\" (UniqueName: \"kubernetes.io/projected/a903ebd7-cd11-4136-ad15-9bd34d3ae235-kube-api-access-sg9xd\") pod \"test-curl-7554-f14a62\" (UID: \"a903ebd7-cd11-4136-ad15-9bd34d3ae235\") " pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:02.330950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.330881    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9xd\" (UniqueName: \"kubernetes.io/projected/a903ebd7-cd11-4136-ad15-9bd34d3ae235-kube-api-access-sg9xd\") pod \"test-curl-7554-f14a62\" (UID: \"a903ebd7-cd11-4136-ad15-9bd34d3ae235\") " pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:02.387850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.387814    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9xd\" (UniqueName: \"kubernetes.io/projected/a903ebd7-cd11-4136-ad15-9bd34d3ae235-kube-api-access-sg9xd\") pod \"test-curl-7554-f14a62\" (UID: \"a903ebd7-cd11-4136-ad15-9bd34d3ae235\") " pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:02.433723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.433689    5551 generic.go:356] "Generic (PLEG): container finished" podID="5804c8d0-5ea5-45b8-b30b-6b723cff5a9f" containerID="28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6" exitCode=137
Oct 31 19:16:02.433862 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.433734    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-7c5beb"
Oct 31 19:16:02.433862 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.433790    5551 scope.go:122] "RemoveContainer" containerID="28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6"
Oct 31 19:16:02.433980 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.433956    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7554-624f63" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" containerName="test-curl-7554-624f63" containerID="cri-o://d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d" gracePeriod=2
Oct 31 19:16:02.444419 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.444404    5551 scope.go:122] "RemoveContainer" containerID="28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6"
Oct 31 19:16:02.444692 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:02.444675    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6\": container with ID starting with 28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6 not found: ID does not exist" containerID="28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6"
Oct 31 19:16:02.444748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.444701    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6"} err="failed to get container status \"28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6\": rpc error: code = NotFound desc = could not find container \"28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6\": container with ID starting with 28e1f6593a138210405c3770d260a3d6ed24e4a0c4e65d8c7dd9ada81a0538a6 not found: ID does not exist"
Oct 31 19:16:02.447903 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.447884    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:02.489834 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:02.488015    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-624f63\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:16:02.538921 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:02.538888    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-7c5beb\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="5804c8d0-5ea5-45b8-b30b-6b723cff5a9f" pod="opendatahub/test-curl-7554-7c5beb"
Oct 31 19:16:02.590429 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:02.590324    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-624f63\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:16:02.837655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.837629    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-f14a62"]
Oct 31 19:16:02.839236 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:16:02.839209    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda903ebd7_cd11_4136_ad15_9bd34d3ae235.slice/crio-d6ea3d196bbf65dcbb1ae27c4770406824432bab1860354e66ccb1f8a2740470 WatchSource:0}: Error finding container d6ea3d196bbf65dcbb1ae27c4770406824432bab1860354e66ccb1f8a2740470: Status 404 returned error can't find the container with id d6ea3d196bbf65dcbb1ae27c4770406824432bab1860354e66ccb1f8a2740470
Oct 31 19:16:02.909570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:02.909543    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5804c8d0-5ea5-45b8-b30b-6b723cff5a9f" path="/var/lib/kubelet/pods/5804c8d0-5ea5-45b8-b30b-6b723cff5a9f/volumes"
Oct 31 19:16:03.442210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:03.442170    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-f14a62" event={"ID":"a903ebd7-cd11-4136-ad15-9bd34d3ae235","Type":"ContainerStarted","Data":"1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20"}
Oct 31 19:16:03.442397 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:03.442221    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-f14a62" event={"ID":"a903ebd7-cd11-4136-ad15-9bd34d3ae235","Type":"ContainerStarted","Data":"d6ea3d196bbf65dcbb1ae27c4770406824432bab1860354e66ccb1f8a2740470"}
Oct 31 19:16:03.554450 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:03.554401    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7554-f14a62" podStartSLOduration=1.554388294 podStartE2EDuration="1.554388294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:03.552674904 +0000 UTC m=+3266.783913195" watchObservedRunningTime="2026-10-31 19:16:03.554388294 +0000 UTC m=+3266.785626584"
Oct 31 19:16:05.065963 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.065891    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:16:05.227111 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.227078    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ccd8e6-9aba-43d3-811d-68d543aa7d71-kube-api-access-q67sz\" (UniqueName: \"kubernetes.io/projected/03ccd8e6-9aba-43d3-811d-68d543aa7d71-kube-api-access-q67sz\") pod \"03ccd8e6-9aba-43d3-811d-68d543aa7d71\" (UID: \"03ccd8e6-9aba-43d3-811d-68d543aa7d71\") "
Oct 31 19:16:05.229478 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.229418    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ccd8e6-9aba-43d3-811d-68d543aa7d71-kube-api-access-q67sz" pod "03ccd8e6-9aba-43d3-811d-68d543aa7d71" (UID: "03ccd8e6-9aba-43d3-811d-68d543aa7d71"). InnerVolumeSpecName "kube-api-access-q67sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:05.309739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.309703    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7554-f14a62"]
Oct 31 19:16:05.312811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.312773    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7554-f14a62"]
Oct 31 19:16:05.328544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.328417    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q67sz\" (UniqueName: \"kubernetes.io/projected/03ccd8e6-9aba-43d3-811d-68d543aa7d71-kube-api-access-q67sz\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:05.447573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.447537    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7554-a632df"]
Oct 31 19:16:05.448404 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.448384    5551 cpu_manager.go:414] "containerMap: removing container" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" containerName="test-curl-7554-f14a62"
Oct 31 19:16:05.448447 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.448408    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" containerName="test-curl-7554-f14a62"
Oct 31 19:16:05.448574 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.448562    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" containerName="test-curl-7554-f14a62"
Oct 31 19:16:05.452313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.452294    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-a632df"
Oct 31 19:16:05.491112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.491081    5551 generic.go:356] "Generic (PLEG): container finished" podID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" containerID="d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d" exitCode=137
Oct 31 19:16:05.491400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.491139    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:16:05.491400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.491266    5551 scope.go:122] "RemoveContainer" containerID="d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d"
Oct 31 19:16:05.491400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.491361    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7554-f14a62" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" containerName="test-curl-7554-f14a62" containerID="cri-o://1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20" gracePeriod=2
Oct 31 19:16:05.506428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.506404    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-a632df"]
Oct 31 19:16:05.513515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.513426    5551 scope.go:122] "RemoveContainer" containerID="d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d"
Oct 31 19:16:05.513966 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:05.513938    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d\": container with ID starting with d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d not found: ID does not exist" containerID="d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d"
Oct 31 19:16:05.514043 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.513976    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d"} err="failed to get container status \"d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d\": rpc error: code = NotFound desc = could not find container \"d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d\": container with ID starting with d9d0c53e32a843dff870035e4b009c9d8b72ca9ca05cc4569fa89cf8503e348d not found: ID does not exist"
Oct 31 19:16:05.534496 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.530927    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6st6\" (UniqueName: \"kubernetes.io/projected/3d2673e0-1e53-40e1-a281-0a4a3a5ed657-kube-api-access-b6st6\") pod \"test-curl-7554-a632df\" (UID: \"3d2673e0-1e53-40e1-a281-0a4a3a5ed657\") " pod="opendatahub/test-curl-7554-a632df"
Oct 31 19:16:05.610883 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:05.610789    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-f14a62\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:05.633482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.632649    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6st6\" (UniqueName: \"kubernetes.io/projected/3d2673e0-1e53-40e1-a281-0a4a3a5ed657-kube-api-access-b6st6\") pod \"test-curl-7554-a632df\" (UID: \"3d2673e0-1e53-40e1-a281-0a4a3a5ed657\") " pod="opendatahub/test-curl-7554-a632df"
Oct 31 19:16:05.661223 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:05.661184    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-624f63\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:16:05.689144 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.689105    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6st6\" (UniqueName: \"kubernetes.io/projected/3d2673e0-1e53-40e1-a281-0a4a3a5ed657-kube-api-access-b6st6\") pod \"test-curl-7554-a632df\" (UID: \"3d2673e0-1e53-40e1-a281-0a4a3a5ed657\") " pod="opendatahub/test-curl-7554-a632df"
Oct 31 19:16:05.764977 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.764932    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-a632df"
Oct 31 19:16:05.934490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:05.934435    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm" podUID="04aca74a-935f-46af-b988-925981022fcd" containerName="main" containerID="cri-o://9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4" gracePeriod=45
Oct 31 19:16:06.164081 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.164032    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-a632df"]
Oct 31 19:16:06.168724 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:16:06.168677    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d2673e0_1e53_40e1_a281_0a4a3a5ed657.slice/crio-5d376a99a90ac8ee2c5f2f46b15558ed3667fdcc366ffe998936fbc9774ec426 WatchSource:0}: Error finding container 5d376a99a90ac8ee2c5f2f46b15558ed3667fdcc366ffe998936fbc9774ec426: Status 404 returned error can't find the container with id 5d376a99a90ac8ee2c5f2f46b15558ed3667fdcc366ffe998936fbc9774ec426
Oct 31 19:16:06.266627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.266596    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:16:06.317164 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:06.317131    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-f14a62\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:06.343129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343100    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-tmp-dir\") pod \"04aca74a-935f-46af-b988-925981022fcd\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") "
Oct 31 19:16:06.343288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343162    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04aca74a-935f-46af-b988-925981022fcd-tls-certs\" (UniqueName: \"kubernetes.io/secret/04aca74a-935f-46af-b988-925981022fcd-tls-certs\") pod \"04aca74a-935f-46af-b988-925981022fcd\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") "
Oct 31 19:16:06.343288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343238    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-dshm\") pod \"04aca74a-935f-46af-b988-925981022fcd\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") "
Oct 31 19:16:06.343410 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343312    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-model-cache\") pod \"04aca74a-935f-46af-b988-925981022fcd\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") "
Oct 31 19:16:06.343410 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343350    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-home\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-home\") pod \"04aca74a-935f-46af-b988-925981022fcd\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") "
Oct 31 19:16:06.343410 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343389    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-tmp-dir" pod "04aca74a-935f-46af-b988-925981022fcd" (UID: "04aca74a-935f-46af-b988-925981022fcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:16:06.343589 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343478    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04aca74a-935f-46af-b988-925981022fcd-kube-api-access-lr2pw\" (UniqueName: \"kubernetes.io/projected/04aca74a-935f-46af-b988-925981022fcd-kube-api-access-lr2pw\") pod \"04aca74a-935f-46af-b988-925981022fcd\" (UID: \"04aca74a-935f-46af-b988-925981022fcd\") "
Oct 31 19:16:06.343743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343682    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-model-cache" pod "04aca74a-935f-46af-b988-925981022fcd" (UID: "04aca74a-935f-46af-b988-925981022fcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:16:06.343743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.343683    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-home" pod "04aca74a-935f-46af-b988-925981022fcd" (UID: "04aca74a-935f-46af-b988-925981022fcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:16:06.344091 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.344056    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:06.344222 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.344096    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:06.344222 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.344110    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:06.345627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.345604    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04aca74a-935f-46af-b988-925981022fcd-tls-certs" pod "04aca74a-935f-46af-b988-925981022fcd" (UID: "04aca74a-935f-46af-b988-925981022fcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:16:06.345735 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.345714    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-dshm" pod "04aca74a-935f-46af-b988-925981022fcd" (UID: "04aca74a-935f-46af-b988-925981022fcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:16:06.345911 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.345885    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aca74a-935f-46af-b988-925981022fcd-kube-api-access-lr2pw" pod "04aca74a-935f-46af-b988-925981022fcd" (UID: "04aca74a-935f-46af-b988-925981022fcd"). InnerVolumeSpecName "kube-api-access-lr2pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:06.367499 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:06.367469    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-624f63\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:16:06.445208 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.445131    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04aca74a-935f-46af-b988-925981022fcd-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:06.445208 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.445158    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04aca74a-935f-46af-b988-925981022fcd-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:06.445208 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.445168    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2pw\" (UniqueName: \"kubernetes.io/projected/04aca74a-935f-46af-b988-925981022fcd-kube-api-access-lr2pw\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:06.513963 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.513924    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-a632df" event={"ID":"3d2673e0-1e53-40e1-a281-0a4a3a5ed657","Type":"ContainerStarted","Data":"180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e"}
Oct 31 19:16:06.514238 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.514202    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-a632df" event={"ID":"3d2673e0-1e53-40e1-a281-0a4a3a5ed657","Type":"ContainerStarted","Data":"5d376a99a90ac8ee2c5f2f46b15558ed3667fdcc366ffe998936fbc9774ec426"}
Oct 31 19:16:06.516753 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.516724    5551 generic.go:356] "Generic (PLEG): container finished" podID="04aca74a-935f-46af-b988-925981022fcd" containerID="9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4" exitCode=0
Oct 31 19:16:06.516873 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.516853    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm" event={"ID":"04aca74a-935f-46af-b988-925981022fcd","Type":"ContainerDied","Data":"9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4"}
Oct 31 19:16:06.516950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.516894    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm" event={"ID":"04aca74a-935f-46af-b988-925981022fcd","Type":"ContainerDied","Data":"c8c4bcc8b3529bbc22141540dd2ede935ef2883c030569741040b9af356077f3"}
Oct 31 19:16:06.516950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.516922    5551 scope.go:122] "RemoveContainer" containerID="9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4"
Oct 31 19:16:06.517123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.517107    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"
Oct 31 19:16:06.533820 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.533114    5551 scope.go:122] "RemoveContainer" containerID="9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4"
Oct 31 19:16:06.533820 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:06.533474    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4\": container with ID starting with 9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4 not found: ID does not exist" containerID="9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4"
Oct 31 19:16:06.533820 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.533515    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4"} err="failed to get container status \"9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4\": rpc error: code = NotFound desc = could not find container \"9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4\": container with ID starting with 9fef10e0574df39dea50ff5b5773b7ad208dfff16f9220ecadbef800771cb2f4 not found: ID does not exist"
Oct 31 19:16:06.565900 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:06.565859    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-624f63\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:16:06.683988 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.683937    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7554-a632df" podStartSLOduration=1.683923885 podStartE2EDuration="1.683923885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:06.681016633 +0000 UTC m=+3269.912254927" watchObservedRunningTime="2026-10-31 19:16:06.683923885 +0000 UTC m=+3269.915162175"
Oct 31 19:16:06.731687 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:06.731604    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-f14a62\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:06.844895 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.844861    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"]
Oct 31 19:16:06.851142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.851115    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4731a/auth-test-model-c4731a-kserve-64799c55bf-bfplm"]
Oct 31 19:16:06.897099 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:06.897065    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-f14a62\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:06.909962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.909926    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" path="/var/lib/kubelet/pods/03ccd8e6-9aba-43d3-811d-68d543aa7d71/volumes"
Oct 31 19:16:06.910421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:06.910403    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04aca74a-935f-46af-b988-925981022fcd" path="/var/lib/kubelet/pods/04aca74a-935f-46af-b988-925981022fcd/volumes"
Oct 31 19:16:06.947527 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:06.947494    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-624f63\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="03ccd8e6-9aba-43d3-811d-68d543aa7d71" pod="opendatahub/test-curl-7554-624f63"
Oct 31 19:16:07.738894 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:07.738865    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:07.765519 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:07.765427    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a903ebd7-cd11-4136-ad15-9bd34d3ae235-kube-api-access-sg9xd\" (UniqueName: \"kubernetes.io/projected/a903ebd7-cd11-4136-ad15-9bd34d3ae235-kube-api-access-sg9xd\") pod \"a903ebd7-cd11-4136-ad15-9bd34d3ae235\" (UID: \"a903ebd7-cd11-4136-ad15-9bd34d3ae235\") "
Oct 31 19:16:07.767682 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:07.767652    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a903ebd7-cd11-4136-ad15-9bd34d3ae235-kube-api-access-sg9xd" pod "a903ebd7-cd11-4136-ad15-9bd34d3ae235" (UID: "a903ebd7-cd11-4136-ad15-9bd34d3ae235"). InnerVolumeSpecName "kube-api-access-sg9xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:07.789908 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:07.789865    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-f14a62\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:07.867086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:07.867058    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg9xd\" (UniqueName: \"kubernetes.io/projected/a903ebd7-cd11-4136-ad15-9bd34d3ae235-kube-api-access-sg9xd\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:08.130259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.130209    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7554-a632df"]
Oct 31 19:16:08.133977 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.133952    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7554-a632df"]
Oct 31 19:16:08.542228 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.542108    5551 generic.go:356] "Generic (PLEG): container finished" podID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" containerID="1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20" exitCode=137
Oct 31 19:16:08.542228 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.542177    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:08.542228 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.542209    5551 scope.go:122] "RemoveContainer" containerID="1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20"
Oct 31 19:16:08.542636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.542585    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7554-a632df" podUID="3d2673e0-1e53-40e1-a281-0a4a3a5ed657" containerName="test-curl-7554-a632df" containerID="cri-o://180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e" gracePeriod=2
Oct 31 19:16:08.553635 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.553613    5551 scope.go:122] "RemoveContainer" containerID="1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20"
Oct 31 19:16:08.553990 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:08.553969    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20\": container with ID starting with 1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20 not found: ID does not exist" containerID="1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20"
Oct 31 19:16:08.554054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.554000    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20"} err="failed to get container status \"1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20\": rpc error: code = NotFound desc = could not find container \"1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20\": container with ID starting with 1c6d7f3bf05f9a3c285625f52a4a92e8c2ffa5d46118c861d414c00c5857ad20 not found: ID does not exist"
Oct 31 19:16:08.593160 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:08.593109    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-f14a62\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:08.643732 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:08.643693    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-a632df\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="3d2673e0-1e53-40e1-a281-0a4a3a5ed657" pod="opendatahub/test-curl-7554-a632df"
Oct 31 19:16:08.694181 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:08.694141    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7554-f14a62\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" pod="opendatahub/test-curl-7554-f14a62"
Oct 31 19:16:08.908841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:08.908805    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a903ebd7-cd11-4136-ad15-9bd34d3ae235" path="/var/lib/kubelet/pods/a903ebd7-cd11-4136-ad15-9bd34d3ae235/volumes"
Oct 31 19:16:10.158894 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.158861    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"]
Oct 31 19:16:10.159421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.159407    5551 cpu_manager.go:414] "containerMap: removing container" podUID="3d2673e0-1e53-40e1-a281-0a4a3a5ed657" containerName="test-curl-7554-a632df"
Oct 31 19:16:10.159527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.159421    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2673e0-1e53-40e1-a281-0a4a3a5ed657" containerName="test-curl-7554-a632df"
Oct 31 19:16:10.159527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.159438    5551 cpu_manager.go:414] "containerMap: removing container" podUID="04aca74a-935f-46af-b988-925981022fcd" containerName="main"
Oct 31 19:16:10.159527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.159445    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04aca74a-935f-46af-b988-925981022fcd" containerName="main"
Oct 31 19:16:10.159628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.159593    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2673e0-1e53-40e1-a281-0a4a3a5ed657" containerName="test-curl-7554-a632df"
Oct 31 19:16:10.159628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.159603    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="04aca74a-935f-46af-b988-925981022fcd" containerName="main"
Oct 31 19:16:10.164452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.164431    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207268    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-credential-socket\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207358    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-certs\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207410    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57vw\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-kube-api-access-p57vw\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207444    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-envoy\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207515    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-data\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207542    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-podinfo\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207585    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istiod-ca-cert\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207664    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-socket\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.212486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.207868    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-token\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.224098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.224054    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9ced6b-openshift-default-dockercfg-j8ns9\""
Oct 31 19:16:10.282554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.282517    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"]
Oct 31 19:16:10.329531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.314402    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-token\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.329841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.329808    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-credential-socket\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.332015 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.331987    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-certs\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.332225 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.332209    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57vw\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-kube-api-access-p57vw\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.332350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.332335    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-envoy\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.332529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.332512    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-data\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.332644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.332628    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-podinfo\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.332785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.332768    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istiod-ca-cert\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.332946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.332933    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-socket\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.333428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.333394    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-socket\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.333659 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.330881    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-credential-socket\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.334170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.334150    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-certs\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.338428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.336945    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-data\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.340275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.340248    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-envoy\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.340898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.340856    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istiod-ca-cert\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.345921 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.345892    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-podinfo\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.385910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.385879    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-token\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.394620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.394594    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57vw\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-kube-api-access-p57vw\") pod \"e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") " pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.478569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.478426    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:10.774151 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.774121    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-a632df"
Oct 31 19:16:10.840296 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.840269    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2673e0-1e53-40e1-a281-0a4a3a5ed657-kube-api-access-b6st6\" (UniqueName: \"kubernetes.io/projected/3d2673e0-1e53-40e1-a281-0a4a3a5ed657-kube-api-access-b6st6\") pod \"3d2673e0-1e53-40e1-a281-0a4a3a5ed657\" (UID: \"3d2673e0-1e53-40e1-a281-0a4a3a5ed657\") "
Oct 31 19:16:10.842432 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.842411    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2673e0-1e53-40e1-a281-0a4a3a5ed657-kube-api-access-b6st6" pod "3d2673e0-1e53-40e1-a281-0a4a3a5ed657" (UID: "3d2673e0-1e53-40e1-a281-0a4a3a5ed657"). InnerVolumeSpecName "kube-api-access-b6st6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:10.908869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.908845    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2673e0-1e53-40e1-a281-0a4a3a5ed657" path="/var/lib/kubelet/pods/3d2673e0-1e53-40e1-a281-0a4a3a5ed657/volumes"
Oct 31 19:16:10.915825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.915798    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"]
Oct 31 19:16:10.917295 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:16:10.917272    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0210728e_0f3c_4d7e_8dc3_ce6892f4bd1e.slice/crio-47ae4660ab60a600ed8fb25c971d62a2cdfcc824b4a825fb87f5ff6cca33b13c WatchSource:0}: Error finding container 47ae4660ab60a600ed8fb25c971d62a2cdfcc824b4a825fb87f5ff6cca33b13c: Status 404 returned error can't find the container with id 47ae4660ab60a600ed8fb25c971d62a2cdfcc824b4a825fb87f5ff6cca33b13c
Oct 31 19:16:10.919940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.919886    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:10.920022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.919972    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:10.920022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.920002    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:10.941978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:10.941959    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6st6\" (UniqueName: \"kubernetes.io/projected/3d2673e0-1e53-40e1-a281-0a4a3a5ed657-kube-api-access-b6st6\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:11.574007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:11.573955    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" event={"ID":"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e","Type":"ContainerStarted","Data":"2d89697186e655f0dedde32782c3400b4a9060cf122ed2d99df0a2ae65c1ea5a"}
Oct 31 19:16:11.574592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:11.574059    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" event={"ID":"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e","Type":"ContainerStarted","Data":"47ae4660ab60a600ed8fb25c971d62a2cdfcc824b4a825fb87f5ff6cca33b13c"}
Oct 31 19:16:11.575513 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:11.575488    5551 generic.go:356] "Generic (PLEG): container finished" podID="3d2673e0-1e53-40e1-a281-0a4a3a5ed657" containerID="180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e" exitCode=137
Oct 31 19:16:11.575648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:11.575577    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-a632df"
Oct 31 19:16:11.575648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:11.575632    5551 scope.go:122] "RemoveContainer" containerID="180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e"
Oct 31 19:16:11.585845 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:11.585819    5551 scope.go:122] "RemoveContainer" containerID="180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e"
Oct 31 19:16:11.586128 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:11.586110    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e\": container with ID starting with 180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e not found: ID does not exist" containerID="180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e"
Oct 31 19:16:11.586197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:11.586142    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e"} err="failed to get container status \"180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e\": rpc error: code = NotFound desc = could not find container \"180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e\": container with ID starting with 180f34f744ce7d8c0212e374868feabaa7537a083cc4362c8cba0d9a79fb664e not found: ID does not exist"
Oct 31 19:16:11.693236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:11.693180    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" podStartSLOduration=1.693164873 podStartE2EDuration="1.693164873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:11.68891275 +0000 UTC m=+3274.920151058" watchObservedRunningTime="2026-10-31 19:16:11.693164873 +0000 UTC m=+3274.924403162"
Oct 31 19:16:12.478728 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:12.478695    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:12.486226 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:12.486194    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:12.584684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:12.584647    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:12.585708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:12.585688    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:16:16.607221 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:16.607185    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:16:32.145762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:32.145733    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"]
Oct 31 19:16:34.708162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:34.708118    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7563-adf1de"]
Oct 31 19:16:34.712497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:34.712447    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-adf1de"
Oct 31 19:16:34.766562 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:34.766542    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-adf1de"]
Oct 31 19:16:34.812211 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:34.812183    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bg2\""
Oct 31 19:16:34.816885 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:34.816847    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdj5\" (UniqueName: \"kubernetes.io/projected/031c08d2-80b6-411d-8992-f001dcae9705-kube-api-access-rhdj5\") pod \"test-curl-7563-adf1de\" (UID: \"031c08d2-80b6-411d-8992-f001dcae9705\") " pod="opendatahub/test-curl-7563-adf1de"
Oct 31 19:16:34.917630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:34.917593    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdj5\" (UniqueName: \"kubernetes.io/projected/031c08d2-80b6-411d-8992-f001dcae9705-kube-api-access-rhdj5\") pod \"test-curl-7563-adf1de\" (UID: \"031c08d2-80b6-411d-8992-f001dcae9705\") " pod="opendatahub/test-curl-7563-adf1de"
Oct 31 19:16:34.974079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:34.973999    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdj5\" (UniqueName: \"kubernetes.io/projected/031c08d2-80b6-411d-8992-f001dcae9705-kube-api-access-rhdj5\") pod \"test-curl-7563-adf1de\" (UID: \"031c08d2-80b6-411d-8992-f001dcae9705\") " pod="opendatahub/test-curl-7563-adf1de"
Oct 31 19:16:35.024606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:35.024571    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-adf1de"
Oct 31 19:16:35.406675 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:35.406548    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-adf1de"]
Oct 31 19:16:35.409367 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:16:35.409336    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod031c08d2_80b6_411d_8992_f001dcae9705.slice/crio-c14278e4af11f57cb8c7f89266274670cc25f252f80ed11bb485096d21540c9e WatchSource:0}: Error finding container c14278e4af11f57cb8c7f89266274670cc25f252f80ed11bb485096d21540c9e: Status 404 returned error can't find the container with id c14278e4af11f57cb8c7f89266274670cc25f252f80ed11bb485096d21540c9e
Oct 31 19:16:35.782034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:35.781936    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-adf1de" event={"ID":"031c08d2-80b6-411d-8992-f001dcae9705","Type":"ContainerStarted","Data":"2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4"}
Oct 31 19:16:35.782034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:35.781989    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-adf1de" event={"ID":"031c08d2-80b6-411d-8992-f001dcae9705","Type":"ContainerStarted","Data":"c14278e4af11f57cb8c7f89266274670cc25f252f80ed11bb485096d21540c9e"}
Oct 31 19:16:35.894914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:35.894862    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7563-adf1de" podStartSLOduration=1.894844113 podStartE2EDuration="1.894844113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:35.893395966 +0000 UTC m=+3299.124634257" watchObservedRunningTime="2026-10-31 19:16:35.894844113 +0000 UTC m=+3299.126082402"
Oct 31 19:16:37.235431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:37.235394    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7563-adf1de"]
Oct 31 19:16:37.239851 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:37.239823    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7563-adf1de"]
Oct 31 19:16:37.797080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:37.797040    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7563-adf1de" podUID="031c08d2-80b6-411d-8992-f001dcae9705" containerName="test-curl-7563-adf1de" containerID="cri-o://2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4" gracePeriod=2
Oct 31 19:16:37.847618 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:37.847577    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7563-adf1de\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="031c08d2-80b6-411d-8992-f001dcae9705" pod="opendatahub/test-curl-7563-adf1de"
Oct 31 19:16:39.276311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.276277    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"]
Oct 31 19:16:39.276840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.276825    5551 cpu_manager.go:414] "containerMap: removing container" podUID="031c08d2-80b6-411d-8992-f001dcae9705" containerName="test-curl-7563-adf1de"
Oct 31 19:16:39.276911 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.276841    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c08d2-80b6-411d-8992-f001dcae9705" containerName="test-curl-7563-adf1de"
Oct 31 19:16:39.276988 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.276978    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c08d2-80b6-411d-8992-f001dcae9705" containerName="test-curl-7563-adf1de"
Oct 31 19:16:39.280559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.280539    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.333579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.333544    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-75c484-openshift-default-dockercfg-wwh64\""
Oct 31 19:16:39.387431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.387382    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-envoy\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.390726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.390692    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-certs\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.391065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.391040    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-data\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.391302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.391280    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-socket\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.391813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.391790    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061ccc30-ae82-496c-8a6b-eeec32fd3867-istiod-ca-cert\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.392062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.392043    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-credential-socket\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.395360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.395338    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-podinfo\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.395606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.395586    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-token\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.396164 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.396131    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5d6x\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-kube-api-access-m5d6x\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.413561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.413518    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"]
Oct 31 19:16:39.501055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501014    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-credential-socket\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501081    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-podinfo\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501116    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-token\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501162    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5d6x\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-kube-api-access-m5d6x\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501449 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501264    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-envoy\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501449 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501401    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-certs\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501471    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-data\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501469    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-credential-socket\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501529    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-socket\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501558    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061ccc30-ae82-496c-8a6b-eeec32fd3867-istiod-ca-cert\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501729    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-certs\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501800    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-data\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.501942 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.501921    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-socket\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.502580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.502557    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061ccc30-ae82-496c-8a6b-eeec32fd3867-istiod-ca-cert\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.504029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.504004    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-envoy\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.504220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.504202    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-podinfo\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.559101 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.559024    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-token\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.559309 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.559284    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5d6x\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-kube-api-access-m5d6x\") pod \"e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") " pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:39.595025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:39.594987    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:40.058271 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.058247    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-adf1de"
Oct 31 19:16:40.086737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.086710    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"]
Oct 31 19:16:40.088676 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:16:40.088645    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061ccc30_ae82_496c_8a6b_eeec32fd3867.slice/crio-8a32c0c750977a860f3774a0b74dd5962996d1fc4cc549f10901f275c7750cd4 WatchSource:0}: Error finding container 8a32c0c750977a860f3774a0b74dd5962996d1fc4cc549f10901f275c7750cd4: Status 404 returned error can't find the container with id 8a32c0c750977a860f3774a0b74dd5962996d1fc4cc549f10901f275c7750cd4
Oct 31 19:16:40.096484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.092546    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:40.096484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.092643    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:40.096484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.092683    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:40.109168 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.109074    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031c08d2-80b6-411d-8992-f001dcae9705-kube-api-access-rhdj5\" (UniqueName: \"kubernetes.io/projected/031c08d2-80b6-411d-8992-f001dcae9705-kube-api-access-rhdj5\") pod \"031c08d2-80b6-411d-8992-f001dcae9705\" (UID: \"031c08d2-80b6-411d-8992-f001dcae9705\") "
Oct 31 19:16:40.114378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.114342    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031c08d2-80b6-411d-8992-f001dcae9705-kube-api-access-rhdj5" pod "031c08d2-80b6-411d-8992-f001dcae9705" (UID: "031c08d2-80b6-411d-8992-f001dcae9705"). InnerVolumeSpecName "kube-api-access-rhdj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:40.210736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.210701    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhdj5\" (UniqueName: \"kubernetes.io/projected/031c08d2-80b6-411d-8992-f001dcae9705-kube-api-access-rhdj5\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:40.355520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.355425    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:16:40.359605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.359580    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:16:40.818476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.818430    5551 generic.go:356] "Generic (PLEG): container finished" podID="031c08d2-80b6-411d-8992-f001dcae9705" containerID="2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4" exitCode=137
Oct 31 19:16:40.818685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.818496    5551 scope.go:122] "RemoveContainer" containerID="2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4"
Oct 31 19:16:40.818685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.818510    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-adf1de"
Oct 31 19:16:40.820407 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.820382    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" event={"ID":"061ccc30-ae82-496c-8a6b-eeec32fd3867","Type":"ContainerStarted","Data":"756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450"}
Oct 31 19:16:40.820549 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.820503    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" event={"ID":"061ccc30-ae82-496c-8a6b-eeec32fd3867","Type":"ContainerStarted","Data":"8a32c0c750977a860f3774a0b74dd5962996d1fc4cc549f10901f275c7750cd4"}
Oct 31 19:16:40.829342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.829325    5551 scope.go:122] "RemoveContainer" containerID="2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4"
Oct 31 19:16:40.829666 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:40.829647    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4\": container with ID starting with 2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4 not found: ID does not exist" containerID="2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4"
Oct 31 19:16:40.829737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.829672    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4"} err="failed to get container status \"2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4\": rpc error: code = NotFound desc = could not find container \"2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4\": container with ID starting with 2b88b5711c37b2a3d00f29eaf3b1f120ef010183f3a2877ea5767a5077f16be4 not found: ID does not exist"
Oct 31 19:16:40.909575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.909542    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031c08d2-80b6-411d-8992-f001dcae9705" path="/var/lib/kubelet/pods/031c08d2-80b6-411d-8992-f001dcae9705/volumes"
Oct 31 19:16:40.937652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:40.937607    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podStartSLOduration=1.93759077 podStartE2EDuration="1.93759077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:40.935680223 +0000 UTC m=+3304.166918512" watchObservedRunningTime="2026-10-31 19:16:40.93759077 +0000 UTC m=+3304.168829060"
Oct 31 19:16:41.595538 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:41.595501    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:41.601543 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:41.601513    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:41.828579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:41.828547    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:41.829516 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:41.829490    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:16:46.605541 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:46.605506    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:16:47.166510 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.166431    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556" podUID="a5a6cbdf-2115-4436-8cd4-1e45851105fb" containerName="main" containerID="cri-o://dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344" gracePeriod=45
Oct 31 19:16:47.246151 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.246101    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556" podUID="a5a6cbdf-2115-4436-8cd4-1e45851105fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": dial tcp 10.134.0.168:8000: connect: connection refused"
Oct 31 19:16:47.458972 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.458844    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:16:47.529599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.527726    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-model-cache\") pod \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") "
Oct 31 19:16:47.529599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.527860    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-dshm\") pod \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") "
Oct 31 19:16:47.529599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.527901    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tls-certs\") pod \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") "
Oct 31 19:16:47.529599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.527975    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a6cbdf-2115-4436-8cd4-1e45851105fb-kube-api-access-7vf9l\" (UniqueName: \"kubernetes.io/projected/a5a6cbdf-2115-4436-8cd4-1e45851105fb-kube-api-access-7vf9l\") pod \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") "
Oct 31 19:16:47.529599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.528011    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-home\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-home\") pod \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") "
Oct 31 19:16:47.529599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.528094    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tmp-dir\") pod \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\" (UID: \"a5a6cbdf-2115-4436-8cd4-1e45851105fb\") "
Oct 31 19:16:47.529599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.529302    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-model-cache" pod "a5a6cbdf-2115-4436-8cd4-1e45851105fb" (UID: "a5a6cbdf-2115-4436-8cd4-1e45851105fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:16:47.537511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.530710    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-home" pod "a5a6cbdf-2115-4436-8cd4-1e45851105fb" (UID: "a5a6cbdf-2115-4436-8cd4-1e45851105fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:16:47.537511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.531797    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tmp-dir" pod "a5a6cbdf-2115-4436-8cd4-1e45851105fb" (UID: "a5a6cbdf-2115-4436-8cd4-1e45851105fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:16:47.560520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.559815    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a6cbdf-2115-4436-8cd4-1e45851105fb-kube-api-access-7vf9l" pod "a5a6cbdf-2115-4436-8cd4-1e45851105fb" (UID: "a5a6cbdf-2115-4436-8cd4-1e45851105fb"). InnerVolumeSpecName "kube-api-access-7vf9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:47.564184 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.564146    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-dshm" pod "a5a6cbdf-2115-4436-8cd4-1e45851105fb" (UID: "a5a6cbdf-2115-4436-8cd4-1e45851105fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:16:47.565850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.565817    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tls-certs" pod "a5a6cbdf-2115-4436-8cd4-1e45851105fb" (UID: "a5a6cbdf-2115-4436-8cd4-1e45851105fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:16:47.629642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.629612    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:47.629642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.629638    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:47.629642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.629648    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:47.630075 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.629656    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a6cbdf-2115-4436-8cd4-1e45851105fb-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:47.630075 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.629666    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vf9l\" (UniqueName: \"kubernetes.io/projected/a5a6cbdf-2115-4436-8cd4-1e45851105fb-kube-api-access-7vf9l\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:47.630075 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.629674    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a6cbdf-2115-4436-8cd4-1e45851105fb-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:47.873810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.873732    5551 generic.go:356] "Generic (PLEG): container finished" podID="a5a6cbdf-2115-4436-8cd4-1e45851105fb" containerID="dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344" exitCode=0
Oct 31 19:16:47.873810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.873792    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"
Oct 31 19:16:47.874019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.873819    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556" event={"ID":"a5a6cbdf-2115-4436-8cd4-1e45851105fb","Type":"ContainerDied","Data":"dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344"}
Oct 31 19:16:47.874019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.873863    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556" event={"ID":"a5a6cbdf-2115-4436-8cd4-1e45851105fb","Type":"ContainerDied","Data":"898cb53daec594c1e0ee034e034f02ad262fc303762255154eeea84d1b449f86"}
Oct 31 19:16:47.874019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.873884    5551 scope.go:122] "RemoveContainer" containerID="dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344"
Oct 31 19:16:47.884159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.884133    5551 scope.go:122] "RemoveContainer" containerID="dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344"
Oct 31 19:16:47.884534 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:16:47.884496    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344\": container with ID starting with dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344 not found: ID does not exist" containerID="dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344"
Oct 31 19:16:47.884624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:47.884541    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344"} err="failed to get container status \"dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344\": rpc error: code = NotFound desc = could not find container \"dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344\": container with ID starting with dce9ab9a99340854ef59c0eb56462491319305b7f1777f91d32271382c5ca344 not found: ID does not exist"
Oct 31 19:16:48.036379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:48.036275    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"]
Oct 31 19:16:48.039850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:48.039824    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bbd8c6/auth-test-model-bbd8c6-kserve-85d9888b89-9v556"]
Oct 31 19:16:48.446731 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:48.446696    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"]
Oct 31 19:16:48.909172 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:48.909132    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a6cbdf-2115-4436-8cd4-1e45851105fb" path="/var/lib/kubelet/pods/a5a6cbdf-2115-4436-8cd4-1e45851105fb/volumes"
Oct 31 19:16:57.116591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:57.116556    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"]
Oct 31 19:16:57.117006 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:57.116762    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy" containerID="cri-o://da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062" gracePeriod=30
Oct 31 19:16:58.452042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.452005    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"]
Oct 31 19:16:58.452623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.452608    5551 cpu_manager.go:414] "containerMap: removing container" podUID="a5a6cbdf-2115-4436-8cd4-1e45851105fb" containerName="main"
Oct 31 19:16:58.452676 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.452624    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a6cbdf-2115-4436-8cd4-1e45851105fb" containerName="main"
Oct 31 19:16:58.452769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.452759    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a6cbdf-2115-4436-8cd4-1e45851105fb" containerName="main"
Oct 31 19:16:58.457575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.457554    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.507521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507487    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-istio-token\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.507702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507542    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-socket\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.507702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507650    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-data\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.507702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507679    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-credential-socket\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.507877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507740    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-certs\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.507933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507870    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5afe2-4109-475b-b762-e7540af02aba-istiod-ca-cert\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.507933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507925    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5afe2-4109-475b-b762-e7540af02aba-istio-podinfo\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.508048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507959    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-envoy\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.508048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.507996    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lm47\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-kube-api-access-7lm47\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.535208 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.535184    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c50ddc-openshift-default-dockercfg-ls68k\""
Oct 31 19:16:58.565822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.565792    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"]
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609249    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-socket\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609377    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-data\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609419    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-credential-socket\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609495    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-certs\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609664    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5afe2-4109-475b-b762-e7540af02aba-istiod-ca-cert\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609823    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5afe2-4109-475b-b762-e7540af02aba-istio-podinfo\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609869    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-envoy\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609898    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-socket\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.609913    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lm47\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-kube-api-access-7lm47\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.610075    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-istio-token\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.610917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.610577    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-certs\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.611593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.611561    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-data\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.611787 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.611762    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-credential-socket\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.612161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.612132    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5afe2-4109-475b-b762-e7540af02aba-istiod-ca-cert\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.627758 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.627697    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-envoy\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.630565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.630531    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5afe2-4109-475b-b762-e7540af02aba-istio-podinfo\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.670976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.670941    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-istio-token\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.671179 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.671149    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lm47\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-kube-api-access-7lm47\") pod \"e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") " pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:58.773612 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:58.773512    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:16:59.206397 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:59.206366    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"]
Oct 31 19:16:59.207555 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:16:59.207529    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f5afe2_4109_475b_b762_e7540af02aba.slice/crio-987ae04d431cb98aeb00220aaa03fb03a6646eda2a4897a17c99edd1b6a317bc WatchSource:0}: Error finding container 987ae04d431cb98aeb00220aaa03fb03a6646eda2a4897a17c99edd1b6a317bc: Status 404 returned error can't find the container with id 987ae04d431cb98aeb00220aaa03fb03a6646eda2a4897a17c99edd1b6a317bc
Oct 31 19:16:59.210246 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:59.210212    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:59.210350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:59.210276    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:59.210350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:59.210302    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:16:59.953201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:59.953156    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj" event={"ID":"f4f5afe2-4109-475b-b762-e7540af02aba","Type":"ContainerStarted","Data":"562a60d1dab6d8ed71cb120da6700cef043257040f1c67292857a9b95e5168f4"}
Oct 31 19:16:59.953593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:16:59.953260    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj" event={"ID":"f4f5afe2-4109-475b-b762-e7540af02aba","Type":"ContainerStarted","Data":"987ae04d431cb98aeb00220aaa03fb03a6646eda2a4897a17c99edd1b6a317bc"}
Oct 31 19:17:00.120650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:00.120597    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj" podStartSLOduration=2.120579874 podStartE2EDuration="2.120579874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:00.118538091 +0000 UTC m=+3323.349776381" watchObservedRunningTime="2026-10-31 19:17:00.120579874 +0000 UTC m=+3323.351818164"
Oct 31 19:17:00.773873 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:00.773840    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:17:00.779590 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:00.779563    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:17:00.960522 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:00.960491    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:17:00.961372 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:00.961355    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:17:02.395405 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.395350    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:17:02.467376 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467349    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-kube-api-access-96g6d\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-kube-api-access-96g6d\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467408    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-data\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467443    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-token\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-token\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467508    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istiod-ca-cert\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467672 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467565    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-credential-socket\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467672 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467587    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-socket\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467672 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467663    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-envoy\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467738    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-podinfo\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467772    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-certs\") pod \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\" (UID: \"b02a43a7-7a19-46a6-8b7d-b60666f9a615\") "
Oct 31 19:17:02.467833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467771    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-data" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:02.468010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467867    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-credential-socket" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:02.468010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467979    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istiod-ca-cert" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:17:02.468010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.467981    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-socket" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:02.468177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.468150    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-certs" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:02.468642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.468618    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.468788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.468649    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.468788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.468665    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.468788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.468679    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.468788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.468693    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.469901 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.469880    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-token" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:02.470028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.469905    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-kube-api-access-96g6d" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "kube-api-access-96g6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:02.470307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.470258    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-envoy" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:02.470613 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.470593    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-podinfo" pod "b02a43a7-7a19-46a6-8b7d-b60666f9a615" (UID: "b02a43a7-7a19-46a6-8b7d-b60666f9a615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:17:02.570287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.570254    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96g6d\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-kube-api-access-96g6d\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.570420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.570292    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.570420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.570308    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.570420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.570321    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02a43a7-7a19-46a6-8b7d-b60666f9a615-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:02.975525 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.975488    5551 generic.go:356] "Generic (PLEG): container finished" podID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerID="da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062" exitCode=0
Oct 31 19:17:02.975703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.975608    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"
Oct 31 19:17:02.975703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.975633    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" event={"ID":"b02a43a7-7a19-46a6-8b7d-b60666f9a615","Type":"ContainerDied","Data":"da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062"}
Oct 31 19:17:02.975703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.975686    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph" event={"ID":"b02a43a7-7a19-46a6-8b7d-b60666f9a615","Type":"ContainerDied","Data":"ce213aaaec67b72e4c9b5be2f7ab8b9a2de90324bfdf69aa91cd927100d6d2b8"}
Oct 31 19:17:02.975876 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.975708    5551 scope.go:122] "RemoveContainer" containerID="da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062"
Oct 31 19:17:02.986808 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.986792    5551 scope.go:122] "RemoveContainer" containerID="884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8"
Oct 31 19:17:02.996353 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.996335    5551 scope.go:122] "RemoveContainer" containerID="da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062"
Oct 31 19:17:02.996714 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:02.996692    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062\": container with ID starting with da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062 not found: ID does not exist" containerID="da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062"
Oct 31 19:17:02.996811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.996722    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062"} err="failed to get container status \"da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062\": rpc error: code = NotFound desc = could not find container \"da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062\": container with ID starting with da35759659801c4dcea683f419e9479ef3825620e9e1bd315bd868eb88170062 not found: ID does not exist"
Oct 31 19:17:02.996811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.996741    5551 scope.go:122] "RemoveContainer" containerID="884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8"
Oct 31 19:17:02.997003 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:02.996984    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8\": container with ID starting with 884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8 not found: ID does not exist" containerID="884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8"
Oct 31 19:17:02.997067 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:02.997013    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8"} err="failed to get container status \"884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8\": rpc error: code = NotFound desc = could not find container \"884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8\": container with ID starting with 884b4f675d76e09dabe55e4060bb6439a4ed484c6dd347a3da9d650e577a3cb8 not found: ID does not exist"
Oct 31 19:17:03.143955 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.143926    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"]
Oct 31 19:17:03.193923 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.193856    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-d9922f-openshift-default-86585f477c-xscph"]
Oct 31 19:17:03.469086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.468992    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv" podUID="bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" containerName="main" containerID="cri-o://1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c" gracePeriod=45
Oct 31 19:17:03.722829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.722768    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:17:03.785267 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.785235    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tmp-dir\") pod \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") "
Oct 31 19:17:03.785441 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.785277    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tls-certs\") pod \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") "
Oct 31 19:17:03.785550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.785487    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-dshm\") pod \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") "
Oct 31 19:17:03.785550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.785523    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-kube-api-access-jlcs9\" (UniqueName: \"kubernetes.io/projected/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-kube-api-access-jlcs9\") pod \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") "
Oct 31 19:17:03.785651 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.785530    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tmp-dir" pod "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" (UID: "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:03.785651 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.785576    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-home\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-home\") pod \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") "
Oct 31 19:17:03.785757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.785670    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-model-cache\") pod \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\" (UID: \"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3\") "
Oct 31 19:17:03.785893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.785866    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-home" pod "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" (UID: "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:03.786178 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.786152    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-model-cache" pod "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" (UID: "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:03.786309 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.786293    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.786352 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.786318    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.786352 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.786333    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.787779 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.787753    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-dshm" pod "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" (UID: "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:03.787779 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.787762    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tls-certs" pod "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" (UID: "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:17:03.787901 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.787802    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-kube-api-access-jlcs9" pod "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" (UID: "bfc3b0ee-40fb-491e-90c9-3403b12f9fc3"). InnerVolumeSpecName "kube-api-access-jlcs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:03.888041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.888006    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.888041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.888035    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.888241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.888046    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlcs9\" (UniqueName: \"kubernetes.io/projected/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3-kube-api-access-jlcs9\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.984575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.984447    5551 generic.go:356] "Generic (PLEG): container finished" podID="bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" containerID="1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c" exitCode=0
Oct 31 19:17:03.984575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.984514    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"
Oct 31 19:17:03.984575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.984485    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv" event={"ID":"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3","Type":"ContainerDied","Data":"1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c"}
Oct 31 19:17:03.984859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.984612    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv" event={"ID":"bfc3b0ee-40fb-491e-90c9-3403b12f9fc3","Type":"ContainerDied","Data":"e5215739b3cf65ec49f8d9633d5b9ec8cbb69ca87e0e02a10d8b4caea802b67d"}
Oct 31 19:17:03.984859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.984634    5551 scope.go:122] "RemoveContainer" containerID="1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c"
Oct 31 19:17:03.995205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.995189    5551 scope.go:122] "RemoveContainer" containerID="1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c"
Oct 31 19:17:03.995440 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:03.995418    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c\": container with ID starting with 1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c not found: ID does not exist" containerID="1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c"
Oct 31 19:17:03.995525 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:03.995449    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c"} err="failed to get container status \"1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c\": rpc error: code = NotFound desc = could not find container \"1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c\": container with ID starting with 1f4e5af71d21d4bd64ae3e321e61680fc1638713b3c1ddfaeaf1ff8f692f8d3c not found: ID does not exist"
Oct 31 19:17:04.148532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:04.148499    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"]
Oct 31 19:17:04.151727 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:04.151705    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d22e36/auth-test-model-d22e36-kserve-7656bbd65c-mqntv"]
Oct 31 19:17:04.909838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:04.909804    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" path="/var/lib/kubelet/pods/b02a43a7-7a19-46a6-8b7d-b60666f9a615/volumes"
Oct 31 19:17:04.910398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:04.910381    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" path="/var/lib/kubelet/pods/bfc3b0ee-40fb-491e-90c9-3403b12f9fc3/volumes"
Oct 31 19:17:16.606297 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:16.606213    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:17:18.867064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:18.867031    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"]
Oct 31 19:17:18.867550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:18.867231    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" containerID="cri-o://ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d" gracePeriod=30
Oct 31 19:17:22.079270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:22.079236    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:17:22.079684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:22.079300    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:17:24.140625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.140567    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:17:24.175657 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.175624    5551 generic.go:356] "Generic (PLEG): container finished" podID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerID="ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d" exitCode=0
Oct 31 19:17:24.175807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.175679    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"
Oct 31 19:17:24.175807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.175700    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" event={"ID":"f8a134d9-fd92-4d5b-9e5b-1476abfd3092","Type":"ContainerDied","Data":"ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d"}
Oct 31 19:17:24.175807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.175777    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n" event={"ID":"f8a134d9-fd92-4d5b-9e5b-1476abfd3092","Type":"ContainerDied","Data":"b0d44398a5423de9e6a2bf664dc0e6ed7ab73c43fe098e47599457ff727bb2e0"}
Oct 31 19:17:24.175807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.175799    5551 scope.go:122] "RemoveContainer" containerID="ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d"
Oct 31 19:17:24.186340 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.186321    5551 scope.go:122] "RemoveContainer" containerID="ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d"
Oct 31 19:17:24.186628 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:24.186607    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d\": container with ID starting with ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d not found: ID does not exist" containerID="ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d"
Oct 31 19:17:24.186711 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.186638    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d"} err="failed to get container status \"ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d\": rpc error: code = NotFound desc = could not find container \"ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d\": container with ID starting with ae935e9e530a1a1442c888e17cf349721abbf0df03e836e609562237243fce8d not found: ID does not exist"
Oct 31 19:17:24.288086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288046    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-token\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-token\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288109    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-kube-api-access-k2x2d\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-kube-api-access-k2x2d\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288155    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-data\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288187    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-socket\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288304    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istiod-ca-cert\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288377    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-podinfo\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288534    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-credential-socket\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288588    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-envoy\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288752 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288653    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-certs\") pod \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\" (UID: \"f8a134d9-fd92-4d5b-9e5b-1476abfd3092\") "
Oct 31 19:17:24.288818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.288781    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istiod-ca-cert" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:17:24.289195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289028    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-credential-socket" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:24.289306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289206    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-socket" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:24.289306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289216    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-certs" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:24.289306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289284    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-data" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:24.289747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289722    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.289830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289753    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.289830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289767    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.289830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289780    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.289830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.289792    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.291716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.291679    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-kube-api-access-k2x2d" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "kube-api-access-k2x2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:24.291917 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.291876    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-envoy" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:24.292018 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.291996    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-podinfo" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:17:24.292888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.292863    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-token" pod "f8a134d9-fd92-4d5b-9e5b-1476abfd3092" (UID: "f8a134d9-fd92-4d5b-9e5b-1476abfd3092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:24.390686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.390590    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.390686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.390624    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2x2d\" (UniqueName: \"kubernetes.io/projected/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-kube-api-access-k2x2d\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.390686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.390639    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.390686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.390652    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a134d9-fd92-4d5b-9e5b-1476abfd3092-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:17:24.411492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.411435    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"]
Oct 31 19:17:24.412049 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412030    5551 cpu_manager.go:414] "containerMap: removing container" podUID="bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" containerName="main"
Oct 31 19:17:24.412128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412051    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" containerName="main"
Oct 31 19:17:24.412128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412076    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy"
Oct 31 19:17:24.412128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412082    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy"
Oct 31 19:17:24.412128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412107    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy"
Oct 31 19:17:24.412128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412112    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy"
Oct 31 19:17:24.412128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412123    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy"
Oct 31 19:17:24.412128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412127    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy"
Oct 31 19:17:24.412529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412212    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc3b0ee-40fb-491e-90c9-3403b12f9fc3" containerName="main"
Oct 31 19:17:24.412529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412221    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy"
Oct 31 19:17:24.412529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.412234    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" containerName="istio-proxy"
Oct 31 19:17:24.426012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.425982    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.471819 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.471785    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"]
Oct 31 19:17:24.478090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.478044    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9ced6b-serving-cert\""
Oct 31 19:17:24.491857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.491830    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvk4\" (UniqueName: \"kubernetes.io/projected/1f2cb926-801b-4706-a01e-c2318b0d35fa-kube-api-access-8nvk4\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.492012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.491873    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.492012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.491901    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.492164 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.492140    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2cb926-801b-4706-a01e-c2318b0d35fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.593752 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.593720    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvk4\" (UniqueName: \"kubernetes.io/projected/1f2cb926-801b-4706-a01e-c2318b0d35fa-kube-api-access-8nvk4\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.593956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.593762    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.593956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.593781    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.593956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.593913    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2cb926-801b-4706-a01e-c2318b0d35fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.594136 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:24.593966    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-serving-cert: secret "maas-api-e2e-extmod-a-9ced6b-serving-cert" not found
Oct 31 19:17:24.594136 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:24.594036    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls podName:1f2cb926-801b-4706-a01e-c2318b0d35fa nodeName:}" failed. No retries permitted until 2026-10-31 19:17:25.094016591 +0000 UTC m=+3348.325254879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls") pod "maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc" (UID: "1f2cb926-801b-4706-a01e-c2318b0d35fa") : secret "maas-api-e2e-extmod-a-9ced6b-serving-cert" not found
Oct 31 19:17:24.594743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.594708    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2cb926-801b-4706-a01e-c2318b0d35fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.596563 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.596545    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.653335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.653238    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"]
Oct 31 19:17:24.671636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.671604    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-2j42n"]
Oct 31 19:17:24.674856 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.674821    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvk4\" (UniqueName: \"kubernetes.io/projected/1f2cb926-801b-4706-a01e-c2318b0d35fa-kube-api-access-8nvk4\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:24.909892 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:24.909795    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a134d9-fd92-4d5b-9e5b-1476abfd3092" path="/var/lib/kubelet/pods/f8a134d9-fd92-4d5b-9e5b-1476abfd3092/volumes"
Oct 31 19:17:25.101651 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:25.101610    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:25.104364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:25.104333    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:25.346332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:25.346249    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:25.738440 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:17:25.738399    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2cb926_801b_4706_a01e_c2318b0d35fa.slice/crio-797fc535be623623bfcc790ea2a646800377a6fae8e16c4050732e30512ada56 WatchSource:0}: Error finding container 797fc535be623623bfcc790ea2a646800377a6fae8e16c4050732e30512ada56: Status 404 returned error can't find the container with id 797fc535be623623bfcc790ea2a646800377a6fae8e16c4050732e30512ada56
Oct 31 19:17:25.742997 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:25.742968    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"]
Oct 31 19:17:26.193128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:26.193091    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc" event={"ID":"1f2cb926-801b-4706-a01e-c2318b0d35fa","Type":"ContainerStarted","Data":"797fc535be623623bfcc790ea2a646800377a6fae8e16c4050732e30512ada56"}
Oct 31 19:17:27.202136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:27.202096    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc" event={"ID":"1f2cb926-801b-4706-a01e-c2318b0d35fa","Type":"ContainerStarted","Data":"5d1fa05ffc82e1fb6534b38c8e11204f4dcbc85198033cabff114069f2880a1d"}
Oct 31 19:17:27.202640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:27.202200    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:27.319094 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:27.319039    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc" podStartSLOduration=2.410169609 podStartE2EDuration="3.319024855s" totalImagesPullingTime="908.855246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:27.315942142 +0000 UTC m=+3350.547180427" watchObservedRunningTime="2026-10-31 19:17:27.319024855 +0000 UTC m=+3350.550263142"
Oct 31 19:17:33.214571 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:33.214535    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:17:46.606840 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:46.606806    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:17:54.167752 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.167697    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"]
Oct 31 19:17:54.168757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.168732    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02a43a7-7a19-46a6-8b7d-b60666f9a615" containerName="istio-proxy"
Oct 31 19:17:54.171531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.171510    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.275243 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.275207    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c50ddc\"/\"openshift-service-ca.crt\""
Oct 31 19:17:54.275243 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.275233    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c50ddc\"/\"kube-root-ca.crt\""
Oct 31 19:17:54.275545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.275231    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c50ddc\"/\"noten-model-c50ddc-kserve-self-signed-certs\""
Oct 31 19:17:54.275545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.275298    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c50ddc\"/\"default-dockercfg-5z7s9\""
Oct 31 19:17:54.279023 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.278997    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"]
Oct 31 19:17:54.308915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.308337    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dzb\" (UniqueName: \"kubernetes.io/projected/4dd8a84e-02d7-47e1-b157-825d18895109-kube-api-access-r7dzb\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.308915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.308403    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-tmp-dir\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.308915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.308503    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-home\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.308915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.308686    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-model-cache\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.311335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.310475    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-dshm\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.311335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.310554    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dd8a84e-02d7-47e1-b157-825d18895109-tls-certs\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.411569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.411532    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-model-cache\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.411762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.411615    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-dshm\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.411762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.411641    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dd8a84e-02d7-47e1-b157-825d18895109-tls-certs\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.411762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.411692    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dzb\" (UniqueName: \"kubernetes.io/projected/4dd8a84e-02d7-47e1-b157-825d18895109-kube-api-access-r7dzb\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.411762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.411725    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-tmp-dir\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.412012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.411794    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-home\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.412102 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.412073    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-model-cache\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.412152 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.412099    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-tmp-dir\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.412152 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.412132    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-home\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.414113 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.414085    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-dshm\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.414292 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.414274    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dd8a84e-02d7-47e1-b157-825d18895109-tls-certs\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.441606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.441543    5551 generic.go:356] "Generic (PLEG): container finished" podID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerID="2d89697186e655f0dedde32782c3400b4a9060cf122ed2d99df0a2ae65c1ea5a" exitCode=0
Oct 31 19:17:54.441706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.441617    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" event={"ID":"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e","Type":"ContainerDied","Data":"2d89697186e655f0dedde32782c3400b4a9060cf122ed2d99df0a2ae65c1ea5a"}
Oct 31 19:17:54.442045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.442029    5551 scope.go:122] "RemoveContainer" containerID="2d89697186e655f0dedde32782c3400b4a9060cf122ed2d99df0a2ae65c1ea5a"
Oct 31 19:17:54.443556 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.443532    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:17:54.443609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.443597    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:17:54.443646 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.443630    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:17:54.470600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.470572    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dzb\" (UniqueName: \"kubernetes.io/projected/4dd8a84e-02d7-47e1-b157-825d18895109-kube-api-access-r7dzb\") pod \"noten-model-c50ddc-kserve-98b9df9f7-9qvsb\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") " pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.490628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.490593    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:54.881731 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:54.881696    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"]
Oct 31 19:17:54.882347 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:17:54.882315    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dd8a84e_02d7_47e1_b157_825d18895109.slice/crio-f16d0380fdf123bfd5dd5a109ab5c3b6d35cdbe819ddcfeb941771df860f8e52 WatchSource:0}: Error finding container f16d0380fdf123bfd5dd5a109ab5c3b6d35cdbe819ddcfeb941771df860f8e52: Status 404 returned error can't find the container with id f16d0380fdf123bfd5dd5a109ab5c3b6d35cdbe819ddcfeb941771df860f8e52
Oct 31 19:17:55.457939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:55.457902    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" event={"ID":"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e","Type":"ContainerStarted","Data":"f5562ac0bf7580a704a5426d47f85190e078ba576e38260614ba36b2dc344425"}
Oct 31 19:17:55.471284 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:55.471244    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb" event={"ID":"4dd8a84e-02d7-47e1-b157-825d18895109","Type":"ContainerStarted","Data":"61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94"}
Oct 31 19:17:55.471587 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:55.471561    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb" event={"ID":"4dd8a84e-02d7-47e1-b157-825d18895109","Type":"ContainerStarted","Data":"f16d0380fdf123bfd5dd5a109ab5c3b6d35cdbe819ddcfeb941771df860f8e52"}
Oct 31 19:17:55.485867 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:55.485831    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:17:55.487099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:55.487068    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:17:55.487253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:55.487126    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 31 19:17:55.493588 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:55.493558    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:17:55.725695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:55.725594    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb" podStartSLOduration=1.725580508 podStartE2EDuration="1.725580508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:55.722356035 +0000 UTC m=+3378.953594324" watchObservedRunningTime="2026-10-31 19:17:55.725580508 +0000 UTC m=+3378.956818797"
Oct 31 19:17:56.479252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:56.479215    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:17:56.479767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:56.479278    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 31 19:17:57.480971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.480922    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:17:57.481550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.480991    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 31 19:17:57.584945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.584905    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:17:57.767229 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.767144    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"]
Oct 31 19:17:57.781361 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.781321    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.794972 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.794947    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2q45\" (UniqueName: \"kubernetes.io/projected/fc13d4d4-721e-4ba8-b7df-ebd258582660-kube-api-access-r2q45\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.795120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.794978    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.795120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.795083    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-metrics-certs\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.795248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.795217    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13d4d4-721e-4ba8-b7df-ebd258582660-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.829776 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.829742    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"]
Oct 31 19:17:57.880473 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.880415    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-75c484-serving-cert\""
Oct 31 19:17:57.896057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.896022    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-metrics-certs\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.896278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.896088    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13d4d4-721e-4ba8-b7df-ebd258582660-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.896278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.896172    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2q45\" (UniqueName: \"kubernetes.io/projected/fc13d4d4-721e-4ba8-b7df-ebd258582660-kube-api-access-r2q45\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.896278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.896193    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.896400 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:57.896329    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-serving-cert: secret "maas-api-e2e-extmod-a-75c484-serving-cert" not found
Oct 31 19:17:57.896400 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:17:57.896382    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls podName:fc13d4d4-721e-4ba8-b7df-ebd258582660 nodeName:}" failed. No retries permitted until 2026-10-31 19:17:58.396366435 +0000 UTC m=+3381.627604703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls") pod "maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj" (UID: "fc13d4d4-721e-4ba8-b7df-ebd258582660") : secret "maas-api-e2e-extmod-a-75c484-serving-cert" not found
Oct 31 19:17:57.897166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.897130    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13d4d4-721e-4ba8-b7df-ebd258582660-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.902028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.899373    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-metrics-certs\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:57.980546 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:57.979213    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2q45\" (UniqueName: \"kubernetes.io/projected/fc13d4d4-721e-4ba8-b7df-ebd258582660-kube-api-access-r2q45\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:58.407084 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:58.407047    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:58.409641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:58.409615    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls\") pod \"maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:58.479145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:58.479094    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:17:58.479356 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:58.479158    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 31 19:17:58.698759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:58.698664    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:17:59.104839 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:59.104809    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"]
Oct 31 19:17:59.106639 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:17:59.106610    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc13d4d4_721e_4ba8_b7df_ebd258582660.slice/crio-7048ccaf6c4adae8d17e39d9c97d90741808b78da479f9fcc9820478adb297db WatchSource:0}: Error finding container 7048ccaf6c4adae8d17e39d9c97d90741808b78da479f9fcc9820478adb297db: Status 404 returned error can't find the container with id 7048ccaf6c4adae8d17e39d9c97d90741808b78da479f9fcc9820478adb297db
Oct 31 19:17:59.479680 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:59.479635    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:17:59.479884 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:59.479706    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 31 19:17:59.517132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:17:59.517092    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj" event={"ID":"fc13d4d4-721e-4ba8-b7df-ebd258582660","Type":"ContainerStarted","Data":"7048ccaf6c4adae8d17e39d9c97d90741808b78da479f9fcc9820478adb297db"}
Oct 31 19:18:00.478822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:00.478789    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:18:00.479279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:00.478841    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 31 19:18:00.526371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:00.526327    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj" event={"ID":"fc13d4d4-721e-4ba8-b7df-ebd258582660","Type":"ContainerStarted","Data":"5baf291d6847deca3b3f8dd9bea0c6a4ce6b60465219f22814e016db33bf28e7"}
Oct 31 19:18:00.526605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:00.526416    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:18:00.650029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:00.649967    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj" podStartSLOduration=2.736040891 podStartE2EDuration="3.649950592s" totalImagesPullingTime="913.909701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:00.647739666 +0000 UTC m=+3383.878977957" watchObservedRunningTime="2026-10-31 19:18:00.649950592 +0000 UTC m=+3383.881188883"
Oct 31 19:18:01.484079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:01.484051    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:18:01.485239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:01.485218    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:18:04.491421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:04.491385    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:18:04.504149 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:04.504123    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:18:04.565662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:04.565631    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:18:06.544893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:06.544858    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:18:12.303503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:12.303449    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"]
Oct 31 19:18:13.268036 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:13.268002    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"]
Oct 31 19:18:13.437082 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:13.437042    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"]
Oct 31 19:18:13.437595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:13.437303    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj" podUID="f4f5afe2-4109-475b-b762-e7540af02aba" containerName="istio-proxy" containerID="cri-o://562a60d1dab6d8ed71cb120da6700cef043257040f1c67292857a9b95e5168f4" gracePeriod=30
Oct 31 19:18:15.963315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:15.963274    5551 patch_prober.go:29] interesting pod/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:18:15.963865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:15.963335    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj" podUID="f4f5afe2-4109-475b-b762-e7540af02aba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:18:16.606618 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:18:16.606578    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:18:18.693131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.692912    5551 generic.go:356] "Generic (PLEG): container finished" podID="f4f5afe2-4109-475b-b762-e7540af02aba" containerID="562a60d1dab6d8ed71cb120da6700cef043257040f1c67292857a9b95e5168f4" exitCode=0
Oct 31 19:18:18.693131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.692938    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj" event={"ID":"f4f5afe2-4109-475b-b762-e7540af02aba","Type":"ContainerDied","Data":"562a60d1dab6d8ed71cb120da6700cef043257040f1c67292857a9b95e5168f4"}
Oct 31 19:18:18.726848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.726824    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:18:18.879470 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.879419    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f5afe2-4109-475b-b762-e7540af02aba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5afe2-4109-475b-b762-e7540af02aba-istiod-ca-cert\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.879673 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.879604    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-certs\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.879673 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.879652    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-kube-api-access-7lm47\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-kube-api-access-7lm47\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.879805 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.879711    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-envoy\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.879805 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.879765    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-data\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.879915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.879834    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f5afe2-4109-475b-b762-e7540af02aba-istiod-ca-cert" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:18:18.879915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.879856    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-credential-socket\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.880025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.879946    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-certs" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:18.880091 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880062    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-istio-token\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.880145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880107    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-socket\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.880145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880124    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-credential-socket" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:18.880263 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880144    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f5afe2-4109-475b-b762-e7540af02aba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5afe2-4109-475b-b762-e7540af02aba-istio-podinfo\") pod \"f4f5afe2-4109-475b-b762-e7540af02aba\" (UID: \"f4f5afe2-4109-475b-b762-e7540af02aba\") "
Oct 31 19:18:18.880263 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880143    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-data" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:18.880372 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880330    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-socket" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:18.880869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880721    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:18.880869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880747    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:18.880869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880763    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:18.880869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880779    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5afe2-4109-475b-b762-e7540af02aba-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:18.880869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.880796    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:18.882431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.882393    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-istio-token" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:18:18.882647 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.882621    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-kube-api-access-7lm47" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "kube-api-access-7lm47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:18:18.882777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.882623    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f5afe2-4109-475b-b762-e7540af02aba-istio-podinfo" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:18:18.882777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.882739    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-envoy" pod "f4f5afe2-4109-475b-b762-e7540af02aba" (UID: "f4f5afe2-4109-475b-b762-e7540af02aba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:18.982748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.982653    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lm47\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-kube-api-access-7lm47\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:18.983529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.983502    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5afe2-4109-475b-b762-e7540af02aba-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:18.983607 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.983557    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5afe2-4109-475b-b762-e7540af02aba-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:18.983607 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:18.983585    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5afe2-4109-475b-b762-e7540af02aba-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:19.702383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:19.702344    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"
Oct 31 19:18:19.702383 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:19.702367    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj" event={"ID":"f4f5afe2-4109-475b-b762-e7540af02aba","Type":"ContainerDied","Data":"987ae04d431cb98aeb00220aaa03fb03a6646eda2a4897a17c99edd1b6a317bc"}
Oct 31 19:18:19.702946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:19.702440    5551 scope.go:122] "RemoveContainer" containerID="562a60d1dab6d8ed71cb120da6700cef043257040f1c67292857a9b95e5168f4"
Oct 31 19:18:19.970556 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:19.957652    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"]
Oct 31 19:18:19.999825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:19.999646    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c50ddc-openshift-default-5f94cd55d7-fhzzj"]
Oct 31 19:18:20.909867 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:20.909829    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f5afe2-4109-475b-b762-e7540af02aba" path="/var/lib/kubelet/pods/f4f5afe2-4109-475b-b762-e7540af02aba/volumes"
Oct 31 19:18:25.779376 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.779339    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7566-e4802a"]
Oct 31 19:18:25.780254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.780229    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f4f5afe2-4109-475b-b762-e7540af02aba" containerName="istio-proxy"
Oct 31 19:18:25.780363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.780255    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f5afe2-4109-475b-b762-e7540af02aba" containerName="istio-proxy"
Oct 31 19:18:25.780543 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.780516    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f5afe2-4109-475b-b762-e7540af02aba" containerName="istio-proxy"
Oct 31 19:18:25.785550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.785529    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:25.815030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.815003    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9r4\" (UniqueName: \"kubernetes.io/projected/6399c3f5-45a9-43c5-934f-675d01e115cd-kube-api-access-qw9r4\") pod \"test-curl-7566-e4802a\" (UID: \"6399c3f5-45a9-43c5-934f-675d01e115cd\") " pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:25.830784 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.830756    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7566-e4802a"]
Oct 31 19:18:25.875136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.875098    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bg2\""
Oct 31 19:18:25.916308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.916274    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9r4\" (UniqueName: \"kubernetes.io/projected/6399c3f5-45a9-43c5-934f-675d01e115cd-kube-api-access-qw9r4\") pod \"test-curl-7566-e4802a\" (UID: \"6399c3f5-45a9-43c5-934f-675d01e115cd\") " pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:25.976264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:25.976233    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9r4\" (UniqueName: \"kubernetes.io/projected/6399c3f5-45a9-43c5-934f-675d01e115cd-kube-api-access-qw9r4\") pod \"test-curl-7566-e4802a\" (UID: \"6399c3f5-45a9-43c5-934f-675d01e115cd\") " pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:26.098806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:26.098727    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:26.514960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:26.514929    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7566-e4802a"]
Oct 31 19:18:26.517060 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:18:26.517033    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6399c3f5_45a9_43c5_934f_675d01e115cd.slice/crio-b1e8b5950157b124db75db60b75eb457c074e3df5ca9b518f49f0d8248c7527b WatchSource:0}: Error finding container b1e8b5950157b124db75db60b75eb457c074e3df5ca9b518f49f0d8248c7527b: Status 404 returned error can't find the container with id b1e8b5950157b124db75db60b75eb457c074e3df5ca9b518f49f0d8248c7527b
Oct 31 19:18:26.758745 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:26.758706    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7566-e4802a" event={"ID":"6399c3f5-45a9-43c5-934f-675d01e115cd","Type":"ContainerStarted","Data":"e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2"}
Oct 31 19:18:26.758968 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:26.758761    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7566-e4802a" event={"ID":"6399c3f5-45a9-43c5-934f-675d01e115cd","Type":"ContainerStarted","Data":"b1e8b5950157b124db75db60b75eb457c074e3df5ca9b518f49f0d8248c7527b"}
Oct 31 19:18:26.878117 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:26.878011    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7566-e4802a" podStartSLOduration=1.877979697 podStartE2EDuration="1.877979697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:26.87479091 +0000 UTC m=+3410.106029201" watchObservedRunningTime="2026-10-31 19:18:26.877979697 +0000 UTC m=+3410.109217990"
Oct 31 19:18:27.325245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.325138    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv" podUID="da38213f-571c-48e0-b67a-bf85962ef952" containerName="main" containerID="cri-o://cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac" gracePeriod=45
Oct 31 19:18:27.334069 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.334032    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv" podUID="da38213f-571c-48e0-b67a-bf85962ef952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": dial tcp 10.134.0.169:8000: connect: connection refused"
Oct 31 19:18:27.624906 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.624878    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:18:27.742190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742140    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-model-cache\") pod \"da38213f-571c-48e0-b67a-bf85962ef952\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") "
Oct 31 19:18:27.742190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742191    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-tmp-dir\") pod \"da38213f-571c-48e0-b67a-bf85962ef952\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") "
Oct 31 19:18:27.742479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742231    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-home\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-home\") pod \"da38213f-571c-48e0-b67a-bf85962ef952\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") "
Oct 31 19:18:27.742538 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742471    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-model-cache" pod "da38213f-571c-48e0-b67a-bf85962ef952" (UID: "da38213f-571c-48e0-b67a-bf85962ef952"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:27.742538 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742500    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da38213f-571c-48e0-b67a-bf85962ef952-tls-certs\" (UniqueName: \"kubernetes.io/secret/da38213f-571c-48e0-b67a-bf85962ef952-tls-certs\") pod \"da38213f-571c-48e0-b67a-bf85962ef952\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") "
Oct 31 19:18:27.742656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742539    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-tmp-dir" pod "da38213f-571c-48e0-b67a-bf85962ef952" (UID: "da38213f-571c-48e0-b67a-bf85962ef952"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:27.742656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742575    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da38213f-571c-48e0-b67a-bf85962ef952-kube-api-access-l29jd\" (UniqueName: \"kubernetes.io/projected/da38213f-571c-48e0-b67a-bf85962ef952-kube-api-access-l29jd\") pod \"da38213f-571c-48e0-b67a-bf85962ef952\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") "
Oct 31 19:18:27.742656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742611    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-dshm\") pod \"da38213f-571c-48e0-b67a-bf85962ef952\" (UID: \"da38213f-571c-48e0-b67a-bf85962ef952\") "
Oct 31 19:18:27.742831 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.742786    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-home" pod "da38213f-571c-48e0-b67a-bf85962ef952" (UID: "da38213f-571c-48e0-b67a-bf85962ef952"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:27.743248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.743221    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:27.743556 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.743270    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:27.743556 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.743286    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:27.746166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.746136    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da38213f-571c-48e0-b67a-bf85962ef952-kube-api-access-l29jd" pod "da38213f-571c-48e0-b67a-bf85962ef952" (UID: "da38213f-571c-48e0-b67a-bf85962ef952"). InnerVolumeSpecName "kube-api-access-l29jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:18:27.746328 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.746298    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-dshm" pod "da38213f-571c-48e0-b67a-bf85962ef952" (UID: "da38213f-571c-48e0-b67a-bf85962ef952"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:27.748897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.748590    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da38213f-571c-48e0-b67a-bf85962ef952-tls-certs" pod "da38213f-571c-48e0-b67a-bf85962ef952" (UID: "da38213f-571c-48e0-b67a-bf85962ef952"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:18:27.783126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.783093    5551 generic.go:356] "Generic (PLEG): container finished" podID="da38213f-571c-48e0-b67a-bf85962ef952" containerID="cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac" exitCode=0
Oct 31 19:18:27.783317 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.783150    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"
Oct 31 19:18:27.783716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.783442    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv" event={"ID":"da38213f-571c-48e0-b67a-bf85962ef952","Type":"ContainerDied","Data":"cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac"}
Oct 31 19:18:27.783716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.783539    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv" event={"ID":"da38213f-571c-48e0-b67a-bf85962ef952","Type":"ContainerDied","Data":"b8c5b509e61408002b3c4993d296d2004cfc7b60f28ef507ce9654214e02d09f"}
Oct 31 19:18:27.783716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.783566    5551 scope.go:122] "RemoveContainer" containerID="cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac"
Oct 31 19:18:27.807780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.807753    5551 scope.go:122] "RemoveContainer" containerID="cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac"
Oct 31 19:18:27.808125 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:18:27.808092    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac\": container with ID starting with cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac not found: ID does not exist" containerID="cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac"
Oct 31 19:18:27.808221 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.808135    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac"} err="failed to get container status \"cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac\": rpc error: code = NotFound desc = could not find container \"cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac\": container with ID starting with cd31b8021ddf0da4be738b902a68e19334e07b05b46a7cfc492ef192c00ce3ac not found: ID does not exist"
Oct 31 19:18:27.845061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.845031    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da38213f-571c-48e0-b67a-bf85962ef952-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:27.845061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.845060    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l29jd\" (UniqueName: \"kubernetes.io/projected/da38213f-571c-48e0-b67a-bf85962ef952-kube-api-access-l29jd\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:27.845277 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.845072    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da38213f-571c-48e0-b67a-bf85962ef952-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:27.870685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.870597    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:18:27.870685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.870651    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:18:27.954791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.954758    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"]
Oct 31 19:18:27.958348 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:27.958324    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/auth-test-model-dbceb9-kserve-65948c7989-qvvvv"]
Oct 31 19:18:28.290876 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.290837    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb" podUID="4dd8a84e-02d7-47e1-b157-825d18895109" containerName="main" containerID="cri-o://61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94" gracePeriod=45
Oct 31 19:18:28.343209 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.343157    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7566-e4802a"]
Oct 31 19:18:28.347435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.347405    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7566-e4802a"]
Oct 31 19:18:28.582881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.582848    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:18:28.656791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.656751    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-dshm\") pod \"4dd8a84e-02d7-47e1-b157-825d18895109\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") "
Oct 31 19:18:28.656956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.656920    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-tmp-dir\") pod \"4dd8a84e-02d7-47e1-b157-825d18895109\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") "
Oct 31 19:18:28.657031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.656959    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-model-cache\") pod \"4dd8a84e-02d7-47e1-b157-825d18895109\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") "
Oct 31 19:18:28.657031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.657027    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd8a84e-02d7-47e1-b157-825d18895109-tls-certs\" (UniqueName: \"kubernetes.io/secret/4dd8a84e-02d7-47e1-b157-825d18895109-tls-certs\") pod \"4dd8a84e-02d7-47e1-b157-825d18895109\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") "
Oct 31 19:18:28.657158 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.657057    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd8a84e-02d7-47e1-b157-825d18895109-kube-api-access-r7dzb\" (UniqueName: \"kubernetes.io/projected/4dd8a84e-02d7-47e1-b157-825d18895109-kube-api-access-r7dzb\") pod \"4dd8a84e-02d7-47e1-b157-825d18895109\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") "
Oct 31 19:18:28.657158 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.657139    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-home\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-home\") pod \"4dd8a84e-02d7-47e1-b157-825d18895109\" (UID: \"4dd8a84e-02d7-47e1-b157-825d18895109\") "
Oct 31 19:18:28.657267 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.657224    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-tmp-dir" pod "4dd8a84e-02d7-47e1-b157-825d18895109" (UID: "4dd8a84e-02d7-47e1-b157-825d18895109"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:28.657385 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.657366    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-model-cache" pod "4dd8a84e-02d7-47e1-b157-825d18895109" (UID: "4dd8a84e-02d7-47e1-b157-825d18895109"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:28.657825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.657776    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:28.657825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.657773    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-home" pod "4dd8a84e-02d7-47e1-b157-825d18895109" (UID: "4dd8a84e-02d7-47e1-b157-825d18895109"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:28.657825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.657802    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:28.659751 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.659725    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-dshm" pod "4dd8a84e-02d7-47e1-b157-825d18895109" (UID: "4dd8a84e-02d7-47e1-b157-825d18895109"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:28.659945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.659921    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd8a84e-02d7-47e1-b157-825d18895109-tls-certs" pod "4dd8a84e-02d7-47e1-b157-825d18895109" (UID: "4dd8a84e-02d7-47e1-b157-825d18895109"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:18:28.660048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.659986    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd8a84e-02d7-47e1-b157-825d18895109-kube-api-access-r7dzb" pod "4dd8a84e-02d7-47e1-b157-825d18895109" (UID: "4dd8a84e-02d7-47e1-b157-825d18895109"). InnerVolumeSpecName "kube-api-access-r7dzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:18:28.760813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.760771    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dd8a84e-02d7-47e1-b157-825d18895109-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:28.760813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.760811    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dzb\" (UniqueName: \"kubernetes.io/projected/4dd8a84e-02d7-47e1-b157-825d18895109-kube-api-access-r7dzb\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:28.761068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.760826    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:28.761068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.760839    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dd8a84e-02d7-47e1-b157-825d18895109-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:28.794187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.794150    5551 generic.go:356] "Generic (PLEG): container finished" podID="4dd8a84e-02d7-47e1-b157-825d18895109" containerID="61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94" exitCode=0
Oct 31 19:18:28.794392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.794214    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"
Oct 31 19:18:28.794392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.794235    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb" event={"ID":"4dd8a84e-02d7-47e1-b157-825d18895109","Type":"ContainerDied","Data":"61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94"}
Oct 31 19:18:28.794392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.794287    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb" event={"ID":"4dd8a84e-02d7-47e1-b157-825d18895109","Type":"ContainerDied","Data":"f16d0380fdf123bfd5dd5a109ab5c3b6d35cdbe819ddcfeb941771df860f8e52"}
Oct 31 19:18:28.794392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.794310    5551 scope.go:122] "RemoveContainer" containerID="61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94"
Oct 31 19:18:28.796288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.796128    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7566-e4802a" podUID="6399c3f5-45a9-43c5-934f-675d01e115cd" containerName="test-curl-7566-e4802a" containerID="cri-o://e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2" gracePeriod=2
Oct 31 19:18:28.807726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.807707    5551 scope.go:122] "RemoveContainer" containerID="61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94"
Oct 31 19:18:28.807999 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:18:28.807978    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94\": container with ID starting with 61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94 not found: ID does not exist" containerID="61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94"
Oct 31 19:18:28.808065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.808009    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94"} err="failed to get container status \"61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94\": rpc error: code = NotFound desc = could not find container \"61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94\": container with ID starting with 61666a1c6162836934770c8a164fea0797b3df04b35c5e6e1335509ce9aadc94 not found: ID does not exist"
Oct 31 19:18:28.899016 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:18:28.898959    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7566-e4802a\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="6399c3f5-45a9-43c5-934f-675d01e115cd" pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:28.911302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:28.911273    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da38213f-571c-48e0-b67a-bf85962ef952" path="/var/lib/kubelet/pods/da38213f-571c-48e0-b67a-bf85962ef952/volumes"
Oct 31 19:18:29.014149 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:29.014118    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"]
Oct 31 19:18:29.042766 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:29.042734    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb"]
Oct 31 19:18:29.555280 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:29.555217    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-c50ddc/noten-model-c50ddc-kserve-98b9df9f7-9qvsb" podUID="4dd8a84e-02d7-47e1-b157-825d18895109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.189:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:18:30.813245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:30.813171    5551 generic.go:356] "Generic (PLEG): container finished" podID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerID="756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450" exitCode=0
Oct 31 19:18:30.813245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:30.813215    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" event={"ID":"061ccc30-ae82-496c-8a6b-eeec32fd3867","Type":"ContainerDied","Data":"756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450"}
Oct 31 19:18:30.813900 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:30.813882    5551 scope.go:122] "RemoveContainer" containerID="756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450"
Oct 31 19:18:30.815877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:30.815840    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:30.815975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:30.815933    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:30.816019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:30.815980    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:30.912649 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:30.912617    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd8a84e-02d7-47e1-b157-825d18895109" path="/var/lib/kubelet/pods/4dd8a84e-02d7-47e1-b157-825d18895109/volumes"
Oct 31 19:18:31.051370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.051271    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:31.093834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.093806    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6399c3f5-45a9-43c5-934f-675d01e115cd-kube-api-access-qw9r4\" (UniqueName: \"kubernetes.io/projected/6399c3f5-45a9-43c5-934f-675d01e115cd-kube-api-access-qw9r4\") pod \"6399c3f5-45a9-43c5-934f-675d01e115cd\" (UID: \"6399c3f5-45a9-43c5-934f-675d01e115cd\") "
Oct 31 19:18:31.096161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.096129    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6399c3f5-45a9-43c5-934f-675d01e115cd-kube-api-access-qw9r4" pod "6399c3f5-45a9-43c5-934f-675d01e115cd" (UID: "6399c3f5-45a9-43c5-934f-675d01e115cd"). InnerVolumeSpecName "kube-api-access-qw9r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:18:31.196003 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.195966    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw9r4\" (UniqueName: \"kubernetes.io/projected/6399c3f5-45a9-43c5-934f-675d01e115cd-kube-api-access-qw9r4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:31.822208 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.822121    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" event={"ID":"061ccc30-ae82-496c-8a6b-eeec32fd3867","Type":"ContainerStarted","Data":"b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49"}
Oct 31 19:18:31.823629 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.823606    5551 generic.go:356] "Generic (PLEG): container finished" podID="6399c3f5-45a9-43c5-934f-675d01e115cd" containerID="e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2" exitCode=137
Oct 31 19:18:31.823755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.823644    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:31.823755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.823664    5551 scope.go:122] "RemoveContainer" containerID="e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2"
Oct 31 19:18:31.834233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.834212    5551 scope.go:122] "RemoveContainer" containerID="e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2"
Oct 31 19:18:31.834555 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:18:31.834534    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2\": container with ID starting with e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2 not found: ID does not exist" containerID="e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2"
Oct 31 19:18:31.834634 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:31.834564    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2"} err="failed to get container status \"e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2\": rpc error: code = NotFound desc = could not find container \"e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2\": container with ID starting with e89ead0a0a32be7d66868a134690e473750ec8d910fa75983dc6f5e619931de2 not found: ID does not exist"
Oct 31 19:18:31.994638 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:18:31.994587    5551 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7566-e4802a\" is forbidden: User \"system:node:ip-10-0-132-99.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-99.us-west-1.compute.internal' and this object" podUID="6399c3f5-45a9-43c5-934f-675d01e115cd" pod="opendatahub/test-curl-7566-e4802a"
Oct 31 19:18:32.595909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:32.595872    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:18:32.596242 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:32.596218    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 31 19:18:32.596302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:32.596283    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 31 19:18:32.908834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:32.908794    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6399c3f5-45a9-43c5-934f-675d01e115cd" path="/var/lib/kubelet/pods/6399c3f5-45a9-43c5-934f-675d01e115cd/volumes"
Oct 31 19:18:33.595494 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:33.595433    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 31 19:18:33.595680 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:33.595525    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 31 19:18:34.595960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:34.595917    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 31 19:18:34.596374 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:34.595986    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 31 19:18:35.595727 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:35.595691    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 31 19:18:35.595929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:35.595744    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 31 19:18:36.595597 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:36.595555    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 31 19:18:36.596043 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:36.595615    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 31 19:18:37.601115 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:37.601084    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:18:37.601584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:37.601568    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:18:37.602414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:37.602393    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:18:38.001477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:38.001426    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"]
Oct 31 19:18:40.361192 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.361154    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"]
Oct 31 19:18:40.362028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362006    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4dd8a84e-02d7-47e1-b157-825d18895109" containerName="main"
Oct 31 19:18:40.362118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362030    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd8a84e-02d7-47e1-b157-825d18895109" containerName="main"
Oct 31 19:18:40.362118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362077    5551 cpu_manager.go:414] "containerMap: removing container" podUID="da38213f-571c-48e0-b67a-bf85962ef952" containerName="main"
Oct 31 19:18:40.362118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362086    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da38213f-571c-48e0-b67a-bf85962ef952" containerName="main"
Oct 31 19:18:40.362118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362108    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6399c3f5-45a9-43c5-934f-675d01e115cd" containerName="test-curl-7566-e4802a"
Oct 31 19:18:40.362118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362116    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6399c3f5-45a9-43c5-934f-675d01e115cd" containerName="test-curl-7566-e4802a"
Oct 31 19:18:40.362379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362282    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd8a84e-02d7-47e1-b157-825d18895109" containerName="main"
Oct 31 19:18:40.362379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362301    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="da38213f-571c-48e0-b67a-bf85962ef952" containerName="main"
Oct 31 19:18:40.362379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.362313    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6399c3f5-45a9-43c5-934f-675d01e115cd" containerName="test-curl-7566-e4802a"
Oct 31 19:18:40.367123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.367102    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.429638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.429546    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9cf479-openshift-default-dockercfg-9j62d\""
Oct 31 19:18:40.480282 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.480234    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"]
Oct 31 19:18:40.538783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.538747    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.538999 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.538789    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.538999 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.538833    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.539140 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.539014    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.539140 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.539066    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.539260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.539154    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwwh\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-kube-api-access-dtwwh\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.539260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.539215    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.539380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.539318    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.539380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.539365    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641090    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641242    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641273    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641325    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641383    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641428    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641609    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwwh\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-kube-api-access-dtwwh\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641836 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641813    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641678    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.641912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641853    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.642029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641926    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.642029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.641975    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.642148 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.642097    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.642282 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.642263    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.643863 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.643844    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.643991 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.643974    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.713973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.713866    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.714107 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.714053    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwwh\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-kube-api-access-dtwwh\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:40.981877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:40.981783    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:41.429900 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:41.429870    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"]
Oct 31 19:18:41.430817 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:18:41.430792    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc020660c_8b71_48a0_a2e8_7b09c6e253cb.slice/crio-f86983c6a7763aed488a4c38ce4f49670192906d782c6479662f4f0f4b656b6f WatchSource:0}: Error finding container f86983c6a7763aed488a4c38ce4f49670192906d782c6479662f4f0f4b656b6f: Status 404 returned error can't find the container with id f86983c6a7763aed488a4c38ce4f49670192906d782c6479662f4f0f4b656b6f
Oct 31 19:18:41.433521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:41.433492    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:41.433602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:41.433553    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:41.433602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:41.433583    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:41.904131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:41.904092    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" event={"ID":"c020660c-8b71-48a0-a2e8-7b09c6e253cb","Type":"ContainerStarted","Data":"693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63"}
Oct 31 19:18:41.904354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:41.904162    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" event={"ID":"c020660c-8b71-48a0-a2e8-7b09c6e253cb","Type":"ContainerStarted","Data":"f86983c6a7763aed488a4c38ce4f49670192906d782c6479662f4f0f4b656b6f"}
Oct 31 19:18:41.982677 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:41.982644    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:42.043682 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:42.043628    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" podStartSLOduration=2.043611841 podStartE2EDuration="2.043611841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:42.042533749 +0000 UTC m=+3425.273772041" watchObservedRunningTime="2026-10-31 19:18:42.043611841 +0000 UTC m=+3425.274850125"
Oct 31 19:18:42.988145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:42.988056    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:43.917868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:43.917837    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:43.918870 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:43.918853    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:18:46.606012 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:18:46.605964    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:18:53.024391 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.024233    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs" podUID="0352efcd-2ada-483a-8e2c-9c38aab157b1" containerName="main" containerID="cri-o://b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c" gracePeriod=45
Oct 31 19:18:53.310097 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.310071    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:18:53.340452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.340423    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0352efcd-2ada-483a-8e2c-9c38aab157b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0352efcd-2ada-483a-8e2c-9c38aab157b1-tls-certs\") pod \"0352efcd-2ada-483a-8e2c-9c38aab157b1\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") "
Oct 31 19:18:53.340611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.340603    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-tmp-dir\") pod \"0352efcd-2ada-483a-8e2c-9c38aab157b1\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") "
Oct 31 19:18:53.340708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.340692    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0352efcd-2ada-483a-8e2c-9c38aab157b1-kube-api-access-8mh7h\" (UniqueName: \"kubernetes.io/projected/0352efcd-2ada-483a-8e2c-9c38aab157b1-kube-api-access-8mh7h\") pod \"0352efcd-2ada-483a-8e2c-9c38aab157b1\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") "
Oct 31 19:18:53.340792 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.340718    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-dshm\") pod \"0352efcd-2ada-483a-8e2c-9c38aab157b1\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") "
Oct 31 19:18:53.340879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.340860    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-home\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-home\") pod \"0352efcd-2ada-483a-8e2c-9c38aab157b1\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") "
Oct 31 19:18:53.340944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.340879    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-tmp-dir" pod "0352efcd-2ada-483a-8e2c-9c38aab157b1" (UID: "0352efcd-2ada-483a-8e2c-9c38aab157b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:53.340944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.340902    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-model-cache\") pod \"0352efcd-2ada-483a-8e2c-9c38aab157b1\" (UID: \"0352efcd-2ada-483a-8e2c-9c38aab157b1\") "
Oct 31 19:18:53.341156 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.341131    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-home" pod "0352efcd-2ada-483a-8e2c-9c38aab157b1" (UID: "0352efcd-2ada-483a-8e2c-9c38aab157b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:53.341366 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.341338    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-model-cache" pod "0352efcd-2ada-483a-8e2c-9c38aab157b1" (UID: "0352efcd-2ada-483a-8e2c-9c38aab157b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:53.341447 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.341434    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:53.341537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.341489    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:53.342902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.342880    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0352efcd-2ada-483a-8e2c-9c38aab157b1-tls-certs" pod "0352efcd-2ada-483a-8e2c-9c38aab157b1" (UID: "0352efcd-2ada-483a-8e2c-9c38aab157b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:18:53.343434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.343413    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-dshm" pod "0352efcd-2ada-483a-8e2c-9c38aab157b1" (UID: "0352efcd-2ada-483a-8e2c-9c38aab157b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:53.343516 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.343440    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0352efcd-2ada-483a-8e2c-9c38aab157b1-kube-api-access-8mh7h" pod "0352efcd-2ada-483a-8e2c-9c38aab157b1" (UID: "0352efcd-2ada-483a-8e2c-9c38aab157b1"). InnerVolumeSpecName "kube-api-access-8mh7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:18:53.442650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.442616    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mh7h\" (UniqueName: \"kubernetes.io/projected/0352efcd-2ada-483a-8e2c-9c38aab157b1-kube-api-access-8mh7h\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:53.442650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.442643    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:53.442650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.442655    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0352efcd-2ada-483a-8e2c-9c38aab157b1-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:53.442882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.442666    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0352efcd-2ada-483a-8e2c-9c38aab157b1-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:18:53.990113 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.990082    5551 generic.go:356] "Generic (PLEG): container finished" podID="0352efcd-2ada-483a-8e2c-9c38aab157b1" containerID="b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c" exitCode=0
Oct 31 19:18:53.990305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.990147    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"
Oct 31 19:18:53.990305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.990176    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs" event={"ID":"0352efcd-2ada-483a-8e2c-9c38aab157b1","Type":"ContainerDied","Data":"b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c"}
Oct 31 19:18:53.990305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.990221    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs" event={"ID":"0352efcd-2ada-483a-8e2c-9c38aab157b1","Type":"ContainerDied","Data":"7d1b72c7ccad9b24c449f5f3e9a2e8c56265510194c3c92f8973d265fd41eb50"}
Oct 31 19:18:53.990305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:53.990241    5551 scope.go:122] "RemoveContainer" containerID="b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c"
Oct 31 19:18:54.001638 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:54.001622    5551 scope.go:122] "RemoveContainer" containerID="b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c"
Oct 31 19:18:54.001901 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:18:54.001873    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c\": container with ID starting with b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c not found: ID does not exist" containerID="b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c"
Oct 31 19:18:54.001976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:54.001915    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c"} err="failed to get container status \"b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c\": rpc error: code = NotFound desc = could not find container \"b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c\": container with ID starting with b75e2d1503758d0d5ad02eb20e7318a9dda45902c61a378755e38f7bd680607c not found: ID does not exist"
Oct 31 19:18:54.160641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:54.160610    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"]
Oct 31 19:18:54.210363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:54.210333    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/auth-test-model-6cfd68-kserve-745f6b9bcc-zjjhs"]
Oct 31 19:18:54.909154 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:54.909120    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0352efcd-2ada-483a-8e2c-9c38aab157b1" path="/var/lib/kubelet/pods/0352efcd-2ada-483a-8e2c-9c38aab157b1/volumes"
Oct 31 19:18:55.113355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.113324    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"]
Oct 31 19:18:55.114110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.114093    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0352efcd-2ada-483a-8e2c-9c38aab157b1" containerName="main"
Oct 31 19:18:55.114162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.114115    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0352efcd-2ada-483a-8e2c-9c38aab157b1" containerName="main"
Oct 31 19:18:55.114253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.114242    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0352efcd-2ada-483a-8e2c-9c38aab157b1" containerName="main"
Oct 31 19:18:55.119060 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.119041    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.202380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.191703    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-8103d4-openshift-default-dockercfg-69pls\""
Oct 31 19:18:55.202949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.201351    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-podinfo\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.203149 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.203130    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-credential-socket\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.203341 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.203322    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istiod-ca-cert\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.203557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.203539    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-token\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.203721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.203706    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-socket\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.203839 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.203824    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-envoy\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.203945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.203932    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-certs\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.204069 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.204054    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-data\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.204166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.204152    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9kpz\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-kube-api-access-g9kpz\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.249657 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.249625    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"]
Oct 31 19:18:55.306479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306088    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istiod-ca-cert\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.306479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306201    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-token\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.306479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306260    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-socket\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.306479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306292    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-envoy\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.306479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306344    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-certs\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.306479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306394    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-data\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.306479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306420    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9kpz\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-kube-api-access-g9kpz\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.306986 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306504    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-podinfo\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.306986 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.306565    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-credential-socket\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.308295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.307113    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-credential-socket\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.308295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.307383    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-socket\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.308295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.307645    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istiod-ca-cert\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.308295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.307790    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-data\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.308295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.308050    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-certs\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.310494 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.310449    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-podinfo\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.314281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.314260    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-envoy\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.378694 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.378662    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-token\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.383293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.383262    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9kpz\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-kube-api-access-g9kpz\") pod \"e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") " pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.533140 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.533043    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:55.978240 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:55.978203    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"]
Oct 31 19:18:56.011395 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:18:56.011369    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c73965_41b1_4e6b_a9e4_f83b798e169d.slice/crio-17ae6605d9d9b0401e7f361222e1eefac1af15effaf6f9f796521b032822a05c WatchSource:0}: Error finding container 17ae6605d9d9b0401e7f361222e1eefac1af15effaf6f9f796521b032822a05c: Status 404 returned error can't find the container with id 17ae6605d9d9b0401e7f361222e1eefac1af15effaf6f9f796521b032822a05c
Oct 31 19:18:56.014150 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:56.014120    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:56.014234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:56.014189    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:56.014234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:56.014217    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:18:57.016358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:57.016321    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" event={"ID":"a2c73965-41b1-4e6b-a9e4-f83b798e169d","Type":"ContainerStarted","Data":"78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1"}
Oct 31 19:18:57.016874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:57.016381    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" event={"ID":"a2c73965-41b1-4e6b-a9e4-f83b798e169d","Type":"ContainerStarted","Data":"17ae6605d9d9b0401e7f361222e1eefac1af15effaf6f9f796521b032822a05c"}
Oct 31 19:18:57.189420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:57.189363    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podStartSLOduration=2.189343966 podStartE2EDuration="2.189343966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:57.186814006 +0000 UTC m=+3440.418052308" watchObservedRunningTime="2026-10-31 19:18:57.189343966 +0000 UTC m=+3440.420582256"
Oct 31 19:18:57.533345 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:57.533311    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:57.539887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:57.539860    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:58.022593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:58.022563    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:58.023609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:58.023591    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:18:59.699886 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.699845    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"]
Oct 31 19:18:59.704632 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.704612    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.761481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.760537    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-150680-openshift-default-dockercfg-twfrg\""
Oct 31 19:18:59.842761 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.842734    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"]
Oct 31 19:18:59.893170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.893135    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-istio-token\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.893568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.893546    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-certs\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.893756 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.893735    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm28b\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-kube-api-access-dm28b\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.893933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.893914    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-credential-socket\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.894154 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.894133    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-data\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.894343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.894327    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-envoy\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.894506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.894491    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa565ee-1255-4389-9776-d686fdf0a579-istio-podinfo\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.894655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.894642    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-socket\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.894825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.894809    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa565ee-1255-4389-9776-d686fdf0a579-istiod-ca-cert\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.999066 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.998968    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa565ee-1255-4389-9776-d686fdf0a579-istiod-ca-cert\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.999250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999168    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-istio-token\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.999250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999203    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-certs\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.999250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999227    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm28b\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-kube-api-access-dm28b\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.999445 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999262    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-credential-socket\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.999445 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999339    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-data\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:18:59.999742 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999716    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-data\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.000627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999755    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-credential-socket\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.000627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999808    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-certs\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.000627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999831    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa565ee-1255-4389-9776-d686fdf0a579-istiod-ca-cert\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.000627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999856    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-envoy\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.000627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999894    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa565ee-1255-4389-9776-d686fdf0a579-istio-podinfo\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.000627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:18:59.999937    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-socket\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.000627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.000253    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-socket\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.002401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.002378    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-envoy\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.003606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.003586    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa565ee-1255-4389-9776-d686fdf0a579-istio-podinfo\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.071138 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.071103    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-istio-token\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.071310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.071172    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm28b\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-kube-api-access-dm28b\") pod \"e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") " pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.319333 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.319234    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:00.819678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.819648    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"]
Oct 31 19:19:00.821009 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:19:00.820983    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa565ee_1255_4389_9776_d686fdf0a579.slice/crio-60f825d9b8bb28c0105feb76a263c13a119c4469e713f21549ce6ac435b4bb6a WatchSource:0}: Error finding container 60f825d9b8bb28c0105feb76a263c13a119c4469e713f21549ce6ac435b4bb6a: Status 404 returned error can't find the container with id 60f825d9b8bb28c0105feb76a263c13a119c4469e713f21549ce6ac435b4bb6a
Oct 31 19:19:00.823825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.823791    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:00.823928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.823868    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:00.823928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:00.823918    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:01.049191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:01.049156    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" event={"ID":"6fa565ee-1255-4389-9776-d686fdf0a579","Type":"ContainerStarted","Data":"4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70"}
Oct 31 19:19:01.049373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:01.049219    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" event={"ID":"6fa565ee-1255-4389-9776-d686fdf0a579","Type":"ContainerStarted","Data":"60f825d9b8bb28c0105feb76a263c13a119c4469e713f21549ce6ac435b4bb6a"}
Oct 31 19:19:01.182025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:01.181958    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podStartSLOduration=2.181937157 podStartE2EDuration="2.181937157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:01.175355667 +0000 UTC m=+3444.406593958" watchObservedRunningTime="2026-10-31 19:19:01.181937157 +0000 UTC m=+3444.413175450"
Oct 31 19:19:01.320039 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:01.319758    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:01.327686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:01.327657    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:02.057757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:02.057722    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:02.058682 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:02.058659    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:19:16.607283 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:19:16.607217    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:19:28.940940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.940906    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"]
Oct 31 19:19:28.946534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.946511    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.985305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.985261    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758ecd9d-9d6a-424c-801d-505ffd50f82f-istiod-ca-cert\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.985642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.985616    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-socket\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.985811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.985795    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-certs\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.988483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.986068    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-data\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.988483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.986173    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-token\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.988483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.986279    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-podinfo\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.988483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.986312    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-envoy\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.988483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.986362    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-kube-api-access-tvlww\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:28.988483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:28.986413    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-credential-socket\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.002511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.002430    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3744b0-openshift-default-dockercfg-qptdz\""
Oct 31 19:19:29.156498 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.150179    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-token\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.172865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.172826    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-podinfo\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.173582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.173534    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-envoy\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.173739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.173601    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-kube-api-access-tvlww\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.173739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.173641    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-credential-socket\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.173739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.173701    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758ecd9d-9d6a-424c-801d-505ffd50f82f-istiod-ca-cert\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.174882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.173746    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-socket\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.174882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.173807    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-certs\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.174882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.172648    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"]
Oct 31 19:19:29.174882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.173983    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-data\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.174882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.174778    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758ecd9d-9d6a-424c-801d-505ffd50f82f-istiod-ca-cert\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.175392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.175366    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-data\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.175528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.175407    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-socket\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.175688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.175669    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-podinfo\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.175796 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.175774    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-certs\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.176030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.176010    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-credential-socket\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.177540 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.177519    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-envoy\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.217261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.217197    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-token\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.235134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.235106    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-kube-api-access-tvlww\") pod \"e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") " pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.274893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.274866    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:29.718771 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.718736    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"]
Oct 31 19:19:29.752215 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:19:29.752184    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758ecd9d_9d6a_424c_801d_505ffd50f82f.slice/crio-4d50e0af5e2aeff5f3195a9fa8f4c7475f9d246222a33bdbc5c531cfc4b1f5f4 WatchSource:0}: Error finding container 4d50e0af5e2aeff5f3195a9fa8f4c7475f9d246222a33bdbc5c531cfc4b1f5f4: Status 404 returned error can't find the container with id 4d50e0af5e2aeff5f3195a9fa8f4c7475f9d246222a33bdbc5c531cfc4b1f5f4
Oct 31 19:19:29.754967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.754935    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:29.755069 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.755005    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:29.755069 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:29.755033    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:30.319842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:30.319801    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" event={"ID":"758ecd9d-9d6a-424c-801d-505ffd50f82f","Type":"ContainerStarted","Data":"e2b476dad947ba6c731449c059c59ac028f86f139c14994d9747385fde2fa13f"}
Oct 31 19:19:30.320330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:30.319857    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" event={"ID":"758ecd9d-9d6a-424c-801d-505ffd50f82f","Type":"ContainerStarted","Data":"4d50e0af5e2aeff5f3195a9fa8f4c7475f9d246222a33bdbc5c531cfc4b1f5f4"}
Oct 31 19:19:30.445305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:30.445251    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" podStartSLOduration=2.445236852 podStartE2EDuration="2.445236852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:30.441365362 +0000 UTC m=+3473.672603651" watchObservedRunningTime="2026-10-31 19:19:30.445236852 +0000 UTC m=+3473.676475142"
Oct 31 19:19:31.275524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:31.275445    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:31.281179 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:31.281151    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:31.327174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:31.327151    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:31.328143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:31.328124    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:19:42.227607 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.227570    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"]
Oct 31 19:19:42.233945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.233919    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.287061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.286992    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-757472-openshift-default-dockercfg-kdlbz\""
Oct 31 19:19:42.341392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.341354    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"]
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.375545    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13659-9053-4e59-90cf-028b00571fba-istio-podinfo\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.375615    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13659-9053-4e59-90cf-028b00571fba-istiod-ca-cert\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.375642    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-certs\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.375765    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-envoy\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.375818    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5fw\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-kube-api-access-bl5fw\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.375960    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-socket\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.376022    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-credential-socket\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.376087    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-data\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.379488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.376170    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-istio-token\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477412    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-socket\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477492    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-credential-socket\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477550    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-data\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477621    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-istio-token\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477669    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13659-9053-4e59-90cf-028b00571fba-istio-podinfo\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477702    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13659-9053-4e59-90cf-028b00571fba-istiod-ca-cert\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477729    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-certs\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477847    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-envoy\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477877    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-socket\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477894    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-credential-socket\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.477939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.477904    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5fw\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-kube-api-access-bl5fw\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.478982 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.478951    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13659-9053-4e59-90cf-028b00571fba-istiod-ca-cert\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.479198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.479176    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-data\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.479482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.479422    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-certs\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.480779 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.480758    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13659-9053-4e59-90cf-028b00571fba-istio-podinfo\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.481635 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.481613    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-envoy\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.536564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.536533    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5fw\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-kube-api-access-bl5fw\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.536960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.536941    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-istio-token\") pod \"e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") " pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.550119 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.550088    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:42.988186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.988155    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"]
Oct 31 19:19:42.993044 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:19:42.993014    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e13659_9053_4e59_90cf_028b00571fba.slice/crio-71602d89da062c54b5096d236a1eb88ba3f4910d1311801e51f150022c1e6fb4 WatchSource:0}: Error finding container 71602d89da062c54b5096d236a1eb88ba3f4910d1311801e51f150022c1e6fb4: Status 404 returned error can't find the container with id 71602d89da062c54b5096d236a1eb88ba3f4910d1311801e51f150022c1e6fb4
Oct 31 19:19:42.996343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.996306    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:42.996476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.996386    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:42.996476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:42.996434    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:19:43.443250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:43.443206    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" event={"ID":"99e13659-9053-4e59-90cf-028b00571fba","Type":"ContainerStarted","Data":"5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502"}
Oct 31 19:19:43.443743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:43.443269    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" event={"ID":"99e13659-9053-4e59-90cf-028b00571fba","Type":"ContainerStarted","Data":"71602d89da062c54b5096d236a1eb88ba3f4910d1311801e51f150022c1e6fb4"}
Oct 31 19:19:43.550227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:43.550191    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:43.564321 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:43.564269    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" podStartSLOduration=1.5642551679999999 podStartE2EDuration="1.564255168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:43.562365834 +0000 UTC m=+3486.793604126" watchObservedRunningTime="2026-10-31 19:19:43.564255168 +0000 UTC m=+3486.795493458"
Oct 31 19:19:44.556408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:44.556380    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:45.457481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:45.457429    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:45.458425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:19:45.458409    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:19:46.605870 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:19:46.605833    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:20:16.605774 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:20:16.605661    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:20:27.112327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.112293    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"]
Oct 31 19:20:27.116403 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.116385    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.168269 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.168240    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-8103d4-serving-cert\""
Oct 31 19:20:27.221262 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.221217    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"]
Oct 31 19:20:27.222119 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.222085    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.222305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.222283    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xg8\" (UniqueName: \"kubernetes.io/projected/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-kube-api-access-66xg8\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.222435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.222413    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-metrics-certs\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.222667 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.222642    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.323781 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.323746    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.324020 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.323827    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.324020 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.323947    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xg8\" (UniqueName: \"kubernetes.io/projected/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-kube-api-access-66xg8\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.324209 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.324027    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-metrics-certs\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.324447 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:20:27.324423    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-serving-cert: secret "maas-api-e2e-extmod-b-8103d4-serving-cert" not found
Oct 31 19:20:27.324574 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:20:27.324548    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls podName:2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6 nodeName:}" failed. No retries permitted until 2026-10-31 19:20:27.824526307 +0000 UTC m=+3531.055764585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls") pod "maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg" (UID: "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6") : secret "maas-api-e2e-extmod-b-8103d4-serving-cert" not found
Oct 31 19:20:27.324762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.324740    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.326868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.326846    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-metrics-certs\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.382028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.381996    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xg8\" (UniqueName: \"kubernetes.io/projected/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-kube-api-access-66xg8\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.833954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.833876    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:27.836423 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:27.836400    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:28.030388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:28.030357    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:28.413875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:28.413843    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"]
Oct 31 19:20:28.415176 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:20:28.415146    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7d7bf7_d6e2_4169_8c6f_b36a93f836b6.slice/crio-1fc1e7014c8546c965a0b327e918ceeb74e93196750869612f2d73373a895918 WatchSource:0}: Error finding container 1fc1e7014c8546c965a0b327e918ceeb74e93196750869612f2d73373a895918: Status 404 returned error can't find the container with id 1fc1e7014c8546c965a0b327e918ceeb74e93196750869612f2d73373a895918
Oct 31 19:20:28.790914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:28.790796    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg" event={"ID":"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6","Type":"ContainerStarted","Data":"1fc1e7014c8546c965a0b327e918ceeb74e93196750869612f2d73373a895918"}
Oct 31 19:20:29.802116 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:29.802078    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg" event={"ID":"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6","Type":"ContainerStarted","Data":"2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b"}
Oct 31 19:20:29.802608 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:29.802148    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:29.924121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:29.924074    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg" podStartSLOduration=2.1352626949999998 podStartE2EDuration="2.924060076s" totalImagesPullingTime="788.797381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:29.918939729 +0000 UTC m=+3533.150178016" watchObservedRunningTime="2026-10-31 19:20:29.924060076 +0000 UTC m=+3533.155298365"
Oct 31 19:20:35.815976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:35.815945    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:20:41.482163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.482129    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"]
Oct 31 19:20:41.485762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.485744    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583263 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583224    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583427 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583382    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583427 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583415    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwntr\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-kube-api-access-vwntr\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583568    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583601    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583620    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583704 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583672    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74636d56-5323-4d54-9251-35dcf230256f-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583793    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74636d56-5323-4d54-9251-35dcf230256f-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.583942 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.583924    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.684813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.684745    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.684813 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.684781    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntr\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-kube-api-access-vwntr\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685051    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685108    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685133    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685195 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685186    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74636d56-5323-4d54-9251-35dcf230256f-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685192    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685269    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74636d56-5323-4d54-9251-35dcf230256f-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685376    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685433    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685498    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685623    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.685860 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685839    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.686080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.685983    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74636d56-5323-4d54-9251-35dcf230256f-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.688186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.688162    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74636d56-5323-4d54-9251-35dcf230256f-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.688350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.688219    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.700668 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.698920    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"]
Oct 31 19:20:41.742415 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.742342    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.742591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.742442    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwntr\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-kube-api-access-vwntr\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.801747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.801702    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:41.911660 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.911627    5551 generic.go:356] "Generic (PLEG): container finished" podID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerID="693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63" exitCode=0
Oct 31 19:20:41.911821 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.911715    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" event={"ID":"c020660c-8b71-48a0-a2e8-7b09c6e253cb","Type":"ContainerDied","Data":"693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63"}
Oct 31 19:20:41.912271 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.912249    5551 scope.go:122] "RemoveContainer" containerID="693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63"
Oct 31 19:20:41.914088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.914061    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:20:41.914191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.914127    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:20:41.914191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:41.914157    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:20:42.246339 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:42.246311    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"]
Oct 31 19:20:42.246655 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:20:42.246622    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74636d56_5323_4d54_9251_35dcf230256f.slice/crio-2a43b1421c2652f12b9507d24fe7b821137b0f205a6733f0a8028160acdfb46b WatchSource:0}: Error finding container 2a43b1421c2652f12b9507d24fe7b821137b0f205a6733f0a8028160acdfb46b: Status 404 returned error can't find the container with id 2a43b1421c2652f12b9507d24fe7b821137b0f205a6733f0a8028160acdfb46b
Oct 31 19:20:42.250387 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:42.250348    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:20:42.250539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:42.250435    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:20:42.250539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:42.250496    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:20:42.923276 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:42.923243    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" event={"ID":"c020660c-8b71-48a0-a2e8-7b09c6e253cb","Type":"ContainerStarted","Data":"cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667"}
Oct 31 19:20:42.924701 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:42.924672    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" event={"ID":"74636d56-5323-4d54-9251-35dcf230256f","Type":"ContainerStarted","Data":"c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5"}
Oct 31 19:20:42.924840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:42.924723    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" event={"ID":"74636d56-5323-4d54-9251-35dcf230256f","Type":"ContainerStarted","Data":"2a43b1421c2652f12b9507d24fe7b821137b0f205a6733f0a8028160acdfb46b"}
Oct 31 19:20:42.986484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:42.985570    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:20:43.190188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:43.190070    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" podStartSLOduration=2.190049157 podStartE2EDuration="2.190049157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:43.167745121 +0000 UTC m=+3546.398983414" watchObservedRunningTime="2026-10-31 19:20:43.190049157 +0000 UTC m=+3546.421287451"
Oct 31 19:20:43.809481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:43.806775    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:43.811973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:43.811599    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:20:43.811973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:43.811661    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 19:20:43.917848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:43.917814    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:20:43.999509 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:43.987558    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:20:43.999509 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:43.987618    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:20:44.806107 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:44.806056    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:20:44.807099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:44.806556    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 19:20:45.003611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:45.003575    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:20:45.004219 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:45.004183    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:20:45.803484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:45.802876    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:20:45.803484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:45.802953    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 19:20:45.984485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:45.982971    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:20:45.984485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:45.983049    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:20:46.636427 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:20:46.636396    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:20:46.826484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:46.817835    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:20:46.826484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:46.817895    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 19:20:46.982542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:46.982072    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:20:46.982542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:46.982133    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:20:47.803896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:47.803858    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:20:47.804560 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:47.804524    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 19:20:47.989481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:47.987208    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:20:47.989481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:47.987270    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:20:48.807104 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:48.807071    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:48.807508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:48.807493    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:48.808256 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:48.808237    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:20:48.990422 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:48.990384    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:20:48.991593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:20:48.991570    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:21:03.099611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:03.099575    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:21:03.100142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:03.099640    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:21:06.192951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.192914    5551 generic.go:356] "Generic (PLEG): container finished" podID="6fa565ee-1255-4389-9776-d686fdf0a579" containerID="4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70" exitCode=0
Oct 31 19:21:06.193541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.192993    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" event={"ID":"6fa565ee-1255-4389-9776-d686fdf0a579","Type":"ContainerDied","Data":"4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70"}
Oct 31 19:21:06.193611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.193539    5551 scope.go:122] "RemoveContainer" containerID="4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70"
Oct 31 19:21:06.195056 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.195032    5551 generic.go:356] "Generic (PLEG): container finished" podID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerID="78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1" exitCode=0
Oct 31 19:21:06.195188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.195114    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" event={"ID":"a2c73965-41b1-4e6b-a9e4-f83b798e169d","Type":"ContainerDied","Data":"78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1"}
Oct 31 19:21:06.195630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.195612    5551 scope.go:122] "RemoveContainer" containerID="78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1"
Oct 31 19:21:06.195817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.195784    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:06.195891 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.195876    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:06.195937 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.195923    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:06.197734 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.197704    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:06.197848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.197787    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:06.197848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:06.197833    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:07.205384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:07.205342    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" event={"ID":"a2c73965-41b1-4e6b-a9e4-f83b798e169d","Type":"ContainerStarted","Data":"12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f"}
Oct 31 19:21:07.207710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:07.207685    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" event={"ID":"6fa565ee-1255-4389-9776-d686fdf0a579","Type":"ContainerStarted","Data":"578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131"}
Oct 31 19:21:07.319835 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:07.319732    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:21:07.320600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:07.320563    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 19:21:07.320755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:07.320645    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 19:21:07.533268 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:07.533183    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:21:07.533429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:07.533390    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:21:07.533494 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:07.533442    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:21:08.320487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:08.319832    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 19:21:08.320487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:08.319903    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 19:21:08.535999 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:08.535963    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:21:08.536330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:08.536280    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:21:09.319973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:09.319938    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 19:21:09.320231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:09.320201    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 19:21:09.534024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:09.533974    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:21:09.534419 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:09.534047    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:21:10.319595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:10.319550    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 19:21:10.319806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:10.319609    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 19:21:10.533975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:10.533938    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:21:10.534240 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:10.533990    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:21:11.320062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:11.320020    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 19:21:11.320278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:11.320082    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 19:21:11.533742 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:11.533686    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:21:11.533933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:11.533762    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:21:12.322748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:12.322706    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 19:21:12.323174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:12.322773    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 19:21:12.533773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:12.533731    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:21:12.533965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:12.533808    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:21:13.022946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:13.022907    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:21:13.323838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:13.323762    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:21:13.324230 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:13.324144    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:21:13.324854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:13.324834    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:21:13.539002 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:13.538974    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:21:13.540013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:13.539996    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:21:16.605856 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:21:16.605820    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:21:40.422502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:40.422442    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:21:40.425816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:40.425790    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:21:46.614722 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:21:46.614681    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:21:47.370028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:47.369649    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:21:47.370028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:47.369726    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:21:51.727286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.727246    5551 generic.go:356] "Generic (PLEG): container finished" podID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerID="e2b476dad947ba6c731449c059c59ac028f86f139c14994d9747385fde2fa13f" exitCode=0
Oct 31 19:21:51.727826 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.727322    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" event={"ID":"758ecd9d-9d6a-424c-801d-505ffd50f82f","Type":"ContainerDied","Data":"e2b476dad947ba6c731449c059c59ac028f86f139c14994d9747385fde2fa13f"}
Oct 31 19:21:51.727914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.727894    5551 scope.go:122] "RemoveContainer" containerID="e2b476dad947ba6c731449c059c59ac028f86f139c14994d9747385fde2fa13f"
Oct 31 19:21:51.729257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.729224    5551 generic.go:356] "Generic (PLEG): container finished" podID="99e13659-9053-4e59-90cf-028b00571fba" containerID="5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502" exitCode=0
Oct 31 19:21:51.729380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.729257    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" event={"ID":"99e13659-9053-4e59-90cf-028b00571fba","Type":"ContainerDied","Data":"5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502"}
Oct 31 19:21:51.729675 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.729658    5551 scope.go:122] "RemoveContainer" containerID="5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502"
Oct 31 19:21:51.729808 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.729783    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:51.729871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.729849    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:51.729929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.729882    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:51.731326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.731300    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:51.731446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.731375    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:51.731446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:51.731429    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:21:52.739212 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:52.739180    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" event={"ID":"758ecd9d-9d6a-424c-801d-505ffd50f82f","Type":"ContainerStarted","Data":"ec8c8c237e17e00c1d1d6f501806bb4c7149e2ecf67e6850a7504197a0761e03"}
Oct 31 19:21:52.742379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:52.742348    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" event={"ID":"99e13659-9053-4e59-90cf-028b00571fba","Type":"ContainerStarted","Data":"d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2"}
Oct 31 19:21:53.276912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:53.276881    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:21:53.277332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:53.277235    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 19:21:53.277544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:53.277521    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 19:21:53.571537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:53.570581    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:21:53.571537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:53.571019    5551 patch_prober.go:29] interesting pod/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 19:21:53.571537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:53.571070    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 19:21:54.280485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:54.275884    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 19:21:54.280485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:54.275962    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 19:21:54.571542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:54.553901    5551 patch_prober.go:29] interesting pod/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 19:21:54.571542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:54.553965    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 19:21:55.285624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:55.285584    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 19:21:55.286717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:55.286271    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 19:21:55.564379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:55.564288    5551 patch_prober.go:29] interesting pod/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 19:21:55.564705 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:55.564675    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 19:21:56.277872 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:56.277835    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 19:21:56.278298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:56.278265    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 19:21:56.557436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:56.553778    5551 patch_prober.go:29] interesting pod/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 19:21:56.557436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:56.553856    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 19:21:57.276550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:57.276495    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 19:21:57.276984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:57.276941    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 19:21:57.555560 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:57.553880    5551 patch_prober.go:29] interesting pod/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 19:21:57.555560 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:57.553944    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 19:21:58.282084 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:58.282044    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:21:58.282594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:58.282380    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:21:58.283176 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:58.283156    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:21:58.555699 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:58.555628    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:21:58.555951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:58.555912    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:21:58.556682 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:21:58.556665    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:22:16.605596 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:22:16.605554    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:22:46.606084 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:22:46.606043    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:23:06.562021 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:06.561986    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"]
Oct 31 19:23:06.562602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:06.562235    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy" containerID="cri-o://d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2" gracePeriod=30
Oct 31 19:23:11.842668 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.842606    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:23:11.947636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947604    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-credential-socket\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.947811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947648    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-istio-token\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-istio-token\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.947811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947693    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-data\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.947811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947757    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-socket\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.947811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947801    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-envoy\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.948059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947852    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e13659-9053-4e59-90cf-028b00571fba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13659-9053-4e59-90cf-028b00571fba-istiod-ca-cert\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.948059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947905    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-credential-socket" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:11.948059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947933    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99e13659-9053-4e59-90cf-028b00571fba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13659-9053-4e59-90cf-028b00571fba-istio-podinfo\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.948059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.947975    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-kube-api-access-bl5fw\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-kube-api-access-bl5fw\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.948059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948024    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-certs\") pod \"99e13659-9053-4e59-90cf-028b00571fba\" (UID: \"99e13659-9053-4e59-90cf-028b00571fba\") "
Oct 31 19:23:11.948338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948074    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-socket" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:11.948338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948088    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-data" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:11.948493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948352    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-certs" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:11.948493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948273    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e13659-9053-4e59-90cf-028b00571fba-istiod-ca-cert" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:23:11.948684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948648    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:11.948684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948674    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:11.948828 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948690    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:11.948828 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948704    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13659-9053-4e59-90cf-028b00571fba-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:11.948828 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.948717    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:11.950360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.950336    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99e13659-9053-4e59-90cf-028b00571fba-istio-podinfo" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:23:11.950475 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.950443    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-kube-api-access-bl5fw" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "kube-api-access-bl5fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:11.950631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.950606    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-envoy" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:11.950695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:11.950624    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-istio-token" pod "99e13659-9053-4e59-90cf-028b00571fba" (UID: "99e13659-9053-4e59-90cf-028b00571fba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:12.050159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.050119    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:12.050159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.050155    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13659-9053-4e59-90cf-028b00571fba-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:12.050418 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.050170    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13659-9053-4e59-90cf-028b00571fba-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:12.050418 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.050185    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5fw\" (UniqueName: \"kubernetes.io/projected/99e13659-9053-4e59-90cf-028b00571fba-kube-api-access-bl5fw\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:12.464024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.463992    5551 generic.go:356] "Generic (PLEG): container finished" podID="99e13659-9053-4e59-90cf-028b00571fba" containerID="d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2" exitCode=0
Oct 31 19:23:12.464215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.464043    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"
Oct 31 19:23:12.464215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.464062    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" event={"ID":"99e13659-9053-4e59-90cf-028b00571fba","Type":"ContainerDied","Data":"d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2"}
Oct 31 19:23:12.464215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.464105    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr" event={"ID":"99e13659-9053-4e59-90cf-028b00571fba","Type":"ContainerDied","Data":"71602d89da062c54b5096d236a1eb88ba3f4910d1311801e51f150022c1e6fb4"}
Oct 31 19:23:12.464215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.464127    5551 scope.go:122] "RemoveContainer" containerID="d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2"
Oct 31 19:23:12.477630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.477574    5551 scope.go:122] "RemoveContainer" containerID="5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502"
Oct 31 19:23:12.490750 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.490731    5551 scope.go:122] "RemoveContainer" containerID="d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2"
Oct 31 19:23:12.491079 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:12.491062    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2\": container with ID starting with d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2 not found: ID does not exist" containerID="d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2"
Oct 31 19:23:12.491131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.491091    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2"} err="failed to get container status \"d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2\": rpc error: code = NotFound desc = could not find container \"d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2\": container with ID starting with d307904a7399e2528d8e3223ee142853aab432948041ffac2e8e85498f1498c2 not found: ID does not exist"
Oct 31 19:23:12.491131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.491110    5551 scope.go:122] "RemoveContainer" containerID="5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502"
Oct 31 19:23:12.491379 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:12.491357    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502\": container with ID starting with 5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502 not found: ID does not exist" containerID="5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502"
Oct 31 19:23:12.491425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.491388    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502"} err="failed to get container status \"5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502\": rpc error: code = NotFound desc = could not find container \"5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502\": container with ID starting with 5d57d7bfaa6beb789c94d120421575c59756e33ddce12756532bbd6faf0ae502 not found: ID does not exist"
Oct 31 19:23:12.636082 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.636047    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"]
Oct 31 19:23:12.639436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.639407    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-757472-openshift-default-6dd5fcbf47-w8pfr"]
Oct 31 19:23:12.910531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:12.910502    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e13659-9053-4e59-90cf-028b00571fba" path="/var/lib/kubelet/pods/99e13659-9053-4e59-90cf-028b00571fba/volumes"
Oct 31 19:23:16.605994 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:16.605958    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:23:17.366640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.366577    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"]
Oct 31 19:23:17.366927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.366817    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc" podUID="1f2cb926-801b-4706-a01e-c2318b0d35fa" containerName="maas-api" containerID="cri-o://5d1fa05ffc82e1fb6534b38c8e11204f4dcbc85198033cabff114069f2880a1d" gracePeriod=30
Oct 31 19:23:17.508663 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.508632    5551 generic.go:356] "Generic (PLEG): container finished" podID="1f2cb926-801b-4706-a01e-c2318b0d35fa" containerID="5d1fa05ffc82e1fb6534b38c8e11204f4dcbc85198033cabff114069f2880a1d" exitCode=0
Oct 31 19:23:17.508825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.508708    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc" event={"ID":"1f2cb926-801b-4706-a01e-c2318b0d35fa","Type":"ContainerDied","Data":"5d1fa05ffc82e1fb6534b38c8e11204f4dcbc85198033cabff114069f2880a1d"}
Oct 31 19:23:17.838777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.838753    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:23:17.983899 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.983274    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls\") pod \"1f2cb926-801b-4706-a01e-c2318b0d35fa\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") "
Oct 31 19:23:17.983899 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.983561    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-metrics-certs\") pod \"1f2cb926-801b-4706-a01e-c2318b0d35fa\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") "
Oct 31 19:23:17.984227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.984055    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2cb926-801b-4706-a01e-c2318b0d35fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2cb926-801b-4706-a01e-c2318b0d35fa-openshift-service-ca\") pod \"1f2cb926-801b-4706-a01e-c2318b0d35fa\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") "
Oct 31 19:23:17.996255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.984400    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2cb926-801b-4706-a01e-c2318b0d35fa-openshift-service-ca" pod "1f2cb926-801b-4706-a01e-c2318b0d35fa" (UID: "1f2cb926-801b-4706-a01e-c2318b0d35fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:23:17.996255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.984486    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2cb926-801b-4706-a01e-c2318b0d35fa-kube-api-access-8nvk4\" (UniqueName: \"kubernetes.io/projected/1f2cb926-801b-4706-a01e-c2318b0d35fa-kube-api-access-8nvk4\") pod \"1f2cb926-801b-4706-a01e-c2318b0d35fa\" (UID: \"1f2cb926-801b-4706-a01e-c2318b0d35fa\") "
Oct 31 19:23:17.996255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.985354    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2cb926-801b-4706-a01e-c2318b0d35fa-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:17.996255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.987133    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2cb926-801b-4706-a01e-c2318b0d35fa-kube-api-access-8nvk4" pod "1f2cb926-801b-4706-a01e-c2318b0d35fa" (UID: "1f2cb926-801b-4706-a01e-c2318b0d35fa"). InnerVolumeSpecName "kube-api-access-8nvk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:17.996255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.987795    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-metrics-certs" pod "1f2cb926-801b-4706-a01e-c2318b0d35fa" (UID: "1f2cb926-801b-4706-a01e-c2318b0d35fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:23:17.999877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:17.999853    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls" pod "1f2cb926-801b-4706-a01e-c2318b0d35fa" (UID: "1f2cb926-801b-4706-a01e-c2318b0d35fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:23:18.087054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.086975    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:18.087054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.087003    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2cb926-801b-4706-a01e-c2318b0d35fa-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:18.087054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.087014    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nvk4\" (UniqueName: \"kubernetes.io/projected/1f2cb926-801b-4706-a01e-c2318b0d35fa-kube-api-access-8nvk4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:18.516502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.516440    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"
Oct 31 19:23:18.516706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.516499    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc" event={"ID":"1f2cb926-801b-4706-a01e-c2318b0d35fa","Type":"ContainerDied","Data":"797fc535be623623bfcc790ea2a646800377a6fae8e16c4050732e30512ada56"}
Oct 31 19:23:18.516706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.516558    5551 scope.go:122] "RemoveContainer" containerID="5d1fa05ffc82e1fb6534b38c8e11204f4dcbc85198033cabff114069f2880a1d"
Oct 31 19:23:18.693287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.693256    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"]
Oct 31 19:23:18.742431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.742403    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ced6b-7658c5d4c8-w6gmc"]
Oct 31 19:23:18.909818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:18.909788    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2cb926-801b-4706-a01e-c2318b0d35fa" path="/var/lib/kubelet/pods/1f2cb926-801b-4706-a01e-c2318b0d35fa/volumes"
Oct 31 19:23:23.413874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:23.413839    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"]
Oct 31 19:23:23.414277 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:23.414087    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy" containerID="cri-o://f5562ac0bf7580a704a5426d47f85190e078ba576e38260614ba36b2dc344425" gracePeriod=30
Oct 31 19:23:28.628928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.628891    5551 generic.go:356] "Generic (PLEG): container finished" podID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerID="f5562ac0bf7580a704a5426d47f85190e078ba576e38260614ba36b2dc344425" exitCode=0
Oct 31 19:23:28.629332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.628958    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" event={"ID":"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e","Type":"ContainerDied","Data":"f5562ac0bf7580a704a5426d47f85190e078ba576e38260614ba36b2dc344425"}
Oct 31 19:23:28.629332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.629046    5551 scope.go:122] "RemoveContainer" containerID="2d89697186e655f0dedde32782c3400b4a9060cf122ed2d99df0a2ae65c1ea5a"
Oct 31 19:23:28.712818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.712793    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:23:28.756627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.756595    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-socket\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.756783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.756721    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-kube-api-access-p57vw\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-kube-api-access-p57vw\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.756850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.756790    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-podinfo\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.756850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.756819    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-certs\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.756850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.756823    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-socket" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:28.757018 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.756853    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istiod-ca-cert\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.757018 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.756929    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-data\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.757120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757027    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-envoy\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.757120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757070    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-token\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-token\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.757120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757103    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-credential-socket\") pod \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\" (UID: \"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e\") "
Oct 31 19:23:28.757284 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757129    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-certs" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:28.757703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757364    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-data" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:28.757703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757638    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-credential-socket" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:28.757880 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757715    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istiod-ca-cert" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:23:28.757950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757931    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:28.758013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757957    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:28.758013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757973    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:28.758013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.757987    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:28.758013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.758001    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:28.759898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.759865    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-token" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:28.759898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.759886    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-podinfo" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:23:28.760054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.759901    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-envoy" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:28.760054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.759931    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-kube-api-access-p57vw" pod "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" (UID: "0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e"). InnerVolumeSpecName "kube-api-access-p57vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:28.859053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.859029    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:28.859053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.859054    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:28.859198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.859064    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57vw\" (UniqueName: \"kubernetes.io/projected/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-kube-api-access-p57vw\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:28.859198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:28.859074    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:29.637409 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:29.637379    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"
Oct 31 19:23:29.637869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:29.637404    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc" event={"ID":"0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e","Type":"ContainerDied","Data":"47ae4660ab60a600ed8fb25c971d62a2cdfcc824b4a825fb87f5ff6cca33b13c"}
Oct 31 19:23:29.637869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:29.637489    5551 scope.go:122] "RemoveContainer" containerID="f5562ac0bf7580a704a5426d47f85190e078ba576e38260614ba36b2dc344425"
Oct 31 19:23:29.874981 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:29.874944    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"]
Oct 31 19:23:29.893434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:29.893358    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ced6b-openshift-default-57c998bdb9-zpjtc"]
Oct 31 19:23:30.909295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:30.909265    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" path="/var/lib/kubelet/pods/0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e/volumes"
Oct 31 19:23:34.154687 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.154654    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"]
Oct 31 19:23:34.155206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.154878    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg" podUID="2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" containerName="maas-api" containerID="cri-o://2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b" gracePeriod=30
Oct 31 19:23:34.453888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.453861    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:23:34.541095 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.541069    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls\") pod \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") "
Oct 31 19:23:34.541243 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.541177    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-metrics-certs\") pod \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") "
Oct 31 19:23:34.541318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.541265    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-kube-api-access-66xg8\" (UniqueName: \"kubernetes.io/projected/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-kube-api-access-66xg8\") pod \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") "
Oct 31 19:23:34.541318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.541295    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-openshift-service-ca\") pod \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\" (UID: \"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6\") "
Oct 31 19:23:34.541713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.541687    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-openshift-service-ca" pod "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" (UID: "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:23:34.541995 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.541958    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:34.543368 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.543343    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-metrics-certs" pod "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" (UID: "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:23:34.543473 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.543364    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls" pod "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" (UID: "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:23:34.543581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.543564    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-kube-api-access-66xg8" pod "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" (UID: "2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6"). InnerVolumeSpecName "kube-api-access-66xg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:34.643421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.643378    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:34.643421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.643423    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66xg8\" (UniqueName: \"kubernetes.io/projected/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-kube-api-access-66xg8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:34.643645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.643441    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:34.684598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.684515    5551 generic.go:356] "Generic (PLEG): container finished" podID="2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" containerID="2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b" exitCode=0
Oct 31 19:23:34.684598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.684563    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"
Oct 31 19:23:34.684800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.684594    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg" event={"ID":"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6","Type":"ContainerDied","Data":"2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b"}
Oct 31 19:23:34.684800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.684651    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg" event={"ID":"2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6","Type":"ContainerDied","Data":"1fc1e7014c8546c965a0b327e918ceeb74e93196750869612f2d73373a895918"}
Oct 31 19:23:34.684800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.684673    5551 scope.go:122] "RemoveContainer" containerID="2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b"
Oct 31 19:23:34.717327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.717303    5551 scope.go:122] "RemoveContainer" containerID="2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b"
Oct 31 19:23:34.717774 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:34.717749    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b\": container with ID starting with 2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b not found: ID does not exist" containerID="2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b"
Oct 31 19:23:34.717864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.717788    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b"} err="failed to get container status \"2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b\": rpc error: code = NotFound desc = could not find container \"2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b\": container with ID starting with 2ae39fc0d83f75eddedbc0bc2a5c1b1b676cf81933409f1aec5c5a45049fdf4b not found: ID does not exist"
Oct 31 19:23:34.849532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.849498    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"]
Oct 31 19:23:34.852364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.852341    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8103d4-7ff465c6c8-rgjpg"]
Oct 31 19:23:34.910346 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:34.910310    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" path="/var/lib/kubelet/pods/2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6/volumes"
Oct 31 19:23:37.745099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:37.745061    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"]
Oct 31 19:23:37.745709 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:37.745277    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy" containerID="cri-o://578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131" gracePeriod=30
Oct 31 19:23:39.194111 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:39.194080    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"]
Oct 31 19:23:39.194617 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:39.194271    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy" containerID="cri-o://12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f" gracePeriod=30
Oct 31 19:23:43.009877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.009807    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:23:43.162435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162396    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-credential-socket\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.162633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162521    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-certs\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.162633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162552    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fa565ee-1255-4389-9776-d686fdf0a579-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa565ee-1255-4389-9776-d686fdf0a579-istio-podinfo\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.162633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162622    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-socket\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.162810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162665    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa565ee-1255-4389-9776-d686fdf0a579-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa565ee-1255-4389-9776-d686fdf0a579-istiod-ca-cert\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.162810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162683    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-credential-socket" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:43.162810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162696    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-kube-api-access-dm28b\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-kube-api-access-dm28b\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.162810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162751    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-data\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.162810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162767    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-certs" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:43.162810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162803    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-envoy\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.163142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162897    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-istio-token\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-istio-token\") pod \"6fa565ee-1255-4389-9776-d686fdf0a579\" (UID: \"6fa565ee-1255-4389-9776-d686fdf0a579\") "
Oct 31 19:23:43.163142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162955    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-socket" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:43.163142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.162975    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-data" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:43.163452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.163158    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa565ee-1255-4389-9776-d686fdf0a579-istiod-ca-cert" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:23:43.163819 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.163794    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.163910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.163825    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.163910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.163842    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.163910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.163856    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa565ee-1255-4389-9776-d686fdf0a579-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.163910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.163869    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.165013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.164986    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fa565ee-1255-4389-9776-d686fdf0a579-istio-podinfo" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:23:43.165209 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.165180    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-kube-api-access-dm28b" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "kube-api-access-dm28b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:43.165320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.165209    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-envoy" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:43.165320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.165230    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-istio-token" pod "6fa565ee-1255-4389-9776-d686fdf0a579" (UID: "6fa565ee-1255-4389-9776-d686fdf0a579"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:43.265042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.264965    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa565ee-1255-4389-9776-d686fdf0a579-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.265042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.264992    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm28b\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-kube-api-access-dm28b\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.265042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.265003    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa565ee-1255-4389-9776-d686fdf0a579-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.265042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.265012    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa565ee-1255-4389-9776-d686fdf0a579-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:43.541592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.541491    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:23:43.541592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.541543    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:23:43.758250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.758216    5551 generic.go:356] "Generic (PLEG): container finished" podID="6fa565ee-1255-4389-9776-d686fdf0a579" containerID="578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131" exitCode=0
Oct 31 19:23:43.758430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.758270    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"
Oct 31 19:23:43.758430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.758303    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" event={"ID":"6fa565ee-1255-4389-9776-d686fdf0a579","Type":"ContainerDied","Data":"578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131"}
Oct 31 19:23:43.758430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.758355    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf" event={"ID":"6fa565ee-1255-4389-9776-d686fdf0a579","Type":"ContainerDied","Data":"60f825d9b8bb28c0105feb76a263c13a119c4469e713f21549ce6ac435b4bb6a"}
Oct 31 19:23:43.758430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.758377    5551 scope.go:122] "RemoveContainer" containerID="578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131"
Oct 31 19:23:43.818174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.818136    5551 scope.go:122] "RemoveContainer" containerID="4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70"
Oct 31 19:23:43.851849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.851807    5551 scope.go:122] "RemoveContainer" containerID="578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131"
Oct 31 19:23:43.852319 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:43.852294    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131\": container with ID starting with 578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131 not found: ID does not exist" containerID="578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131"
Oct 31 19:23:43.852431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.852335    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131"} err="failed to get container status \"578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131\": rpc error: code = NotFound desc = could not find container \"578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131\": container with ID starting with 578c1a41fbce1764cf60920107a7dfad1b3a5e526d3692ac311139c8d411d131 not found: ID does not exist"
Oct 31 19:23:43.852431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.852361    5551 scope.go:122] "RemoveContainer" containerID="4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70"
Oct 31 19:23:43.853285 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:43.853201    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70\": container with ID starting with 4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70 not found: ID does not exist" containerID="4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70"
Oct 31 19:23:43.853285 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.853236    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70"} err="failed to get container status \"4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70\": rpc error: code = NotFound desc = could not find container \"4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70\": container with ID starting with 4561e82192c6987fd107796366e3d76000d9c2171bdbf2c23ae63a38b5c53f70 not found: ID does not exist"
Oct 31 19:23:43.924401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.924366    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"]
Oct 31 19:23:43.978281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:43.978250    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-150680-openshift-default-6c8b774c7c-bd7jf"]
Oct 31 19:23:44.573510 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.573488    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:23:44.698135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698105    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-data\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.698378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698144    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-socket\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.698378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698204    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-envoy\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.698378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698244    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-podinfo\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.698378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698279    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istiod-ca-cert\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.698378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698370    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-certs\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.698727 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698406    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-data" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:44.698727 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698517    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-credential-socket\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.698727 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698551    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-token\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-token\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.698727 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.698584    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-kube-api-access-g9kpz\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-kube-api-access-g9kpz\") pod \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\" (UID: \"a2c73965-41b1-4e6b-a9e4-f83b798e169d\") "
Oct 31 19:23:44.699331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.699009    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-certs" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:44.699331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.699073    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-socket" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:44.699331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.699184    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-credential-socket" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:44.699331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.699283    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.699331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.699302    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.699331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.699318    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.699331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.699332    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.699751 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.699337    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istiod-ca-cert" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:23:44.701874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.701838    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-token" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:44.701874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.701861    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-envoy" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:44.702058 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.701912    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-kube-api-access-g9kpz" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "kube-api-access-g9kpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:44.702842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.702816    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-podinfo" pod "a2c73965-41b1-4e6b-a9e4-f83b798e169d" (UID: "a2c73965-41b1-4e6b-a9e4-f83b798e169d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:23:44.768019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.767987    5551 generic.go:356] "Generic (PLEG): container finished" podID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerID="12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f" exitCode=0
Oct 31 19:23:44.768162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.768052    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"
Oct 31 19:23:44.768162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.768064    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" event={"ID":"a2c73965-41b1-4e6b-a9e4-f83b798e169d","Type":"ContainerDied","Data":"12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f"}
Oct 31 19:23:44.768162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.768114    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc" event={"ID":"a2c73965-41b1-4e6b-a9e4-f83b798e169d","Type":"ContainerDied","Data":"17ae6605d9d9b0401e7f361222e1eefac1af15effaf6f9f796521b032822a05c"}
Oct 31 19:23:44.768162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.768138    5551 scope.go:122] "RemoveContainer" containerID="12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f"
Oct 31 19:23:44.781162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.781144    5551 scope.go:122] "RemoveContainer" containerID="78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1"
Oct 31 19:23:44.790789 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.790768    5551 scope.go:122] "RemoveContainer" containerID="12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f"
Oct 31 19:23:44.791093 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:44.791063    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f\": container with ID starting with 12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f not found: ID does not exist" containerID="12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f"
Oct 31 19:23:44.791188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.791090    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f"} err="failed to get container status \"12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f\": rpc error: code = NotFound desc = could not find container \"12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f\": container with ID starting with 12c0eed7c4ede9774047df16c9cd2e7775e8037910264f8d0024755b1197580f not found: ID does not exist"
Oct 31 19:23:44.791188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.791108    5551 scope.go:122] "RemoveContainer" containerID="78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1"
Oct 31 19:23:44.791397 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:44.791379    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1\": container with ID starting with 78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1 not found: ID does not exist" containerID="78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1"
Oct 31 19:23:44.791508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.791404    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1"} err="failed to get container status \"78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1\": rpc error: code = NotFound desc = could not find container \"78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1\": container with ID starting with 78795cda0f4c891fead313306651a00a54e8638f28038fec72d5417014920ea1 not found: ID does not exist"
Oct 31 19:23:44.800508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.800416    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.800508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.800450    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.800508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.800482    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.800508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.800496    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9kpz\" (UniqueName: \"kubernetes.io/projected/a2c73965-41b1-4e6b-a9e4-f83b798e169d-kube-api-access-g9kpz\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.800696 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.800510    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c73965-41b1-4e6b-a9e4-f83b798e169d-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:44.909264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.909234    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" path="/var/lib/kubelet/pods/6fa565ee-1255-4389-9776-d686fdf0a579/volumes"
Oct 31 19:23:44.933751 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.933719    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"]
Oct 31 19:23:44.938762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:44.938738    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8103d4-openshift-default-5884f85b97-rl7tc"]
Oct 31 19:23:45.726428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.726394    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"]
Oct 31 19:23:45.726972 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.726956    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy"
Oct 31 19:23:45.726972 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.726972    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.726987    5551 cpu_manager.go:414] "containerMap: removing container" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.726992    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727004    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727009    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727015    5551 cpu_manager.go:414] "containerMap: removing container" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727019    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727035    5551 cpu_manager.go:414] "containerMap: removing container" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727040    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727047    5551 cpu_manager.go:414] "containerMap: removing container" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727051    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727066    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" containerName="maas-api"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727070    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" containerName="maas-api"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727077    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727081    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727095    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1f2cb926-801b-4706-a01e-c2318b0d35fa" containerName="maas-api"
Oct 31 19:23:45.727110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727099    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2cb926-801b-4706-a01e-c2318b0d35fa" containerName="maas-api"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727182    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2cb926-801b-4706-a01e-c2318b0d35fa" containerName="maas-api"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727191    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727197    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727205    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727211    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="0210728e-0f3c-4d7e-8dc3-ce6892f4bd1e" containerName="istio-proxy"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727218    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e13659-9053-4e59-90cf-028b00571fba" containerName="istio-proxy"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727224    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727231    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7d7bf7-d6e2-4169-8c6f-b36a93f836b6" containerName="maas-api"
Oct 31 19:23:45.727922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.727237    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa565ee-1255-4389-9776-d686fdf0a579" containerName="istio-proxy"
Oct 31 19:23:45.732125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.732104    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.784306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.784272    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c2025e74c640\""
Oct 31 19:23:45.784504 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.784391    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 19:23:45.784605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.784539    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 19:23:45.784605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.784588    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 19:23:45.784724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.784705    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 19:23:45.788413 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.788388    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"]
Oct 31 19:23:45.817719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.817680    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-routing-overlay-volume\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.817910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.817888    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-provider-credentials\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.818086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.818060    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5g7\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-kube-api-access-hv5g7\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.818261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.818241    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-plugins-config-volume\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.919942 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.919912    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-provider-credentials\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.920116 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.919979    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5g7\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-kube-api-access-hv5g7\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.920116 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.920047    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-plugins-config-volume\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.920116 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.920073    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-routing-overlay-volume\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.920731 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.920703    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-routing-overlay-volume\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.920869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.920809    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-plugins-config-volume\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.922502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.922484    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-provider-credentials\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:45.977652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:45.977589    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5g7\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-kube-api-access-hv5g7\") pod \"payload-processing-external-model-84869d9799-8gxx8\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") " pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:46.046530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:46.046490    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:46.446658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:46.446625    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"]
Oct 31 19:23:46.449591 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:23:46.449556    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ca7069_74e3_48c7_9825_37761bdc3b08.slice/crio-fd9669663b022554e5fa99353993a2294e67caa05b91f2056fac7dc430bd6fce WatchSource:0}: Error finding container fd9669663b022554e5fa99353993a2294e67caa05b91f2056fac7dc430bd6fce: Status 404 returned error can't find the container with id fd9669663b022554e5fa99353993a2294e67caa05b91f2056fac7dc430bd6fce
Oct 31 19:23:46.604722 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:23:46.604679    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:23:46.786281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:46.786187    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8" event={"ID":"89ca7069-74e3-48c7-9825-37761bdc3b08","Type":"ContainerStarted","Data":"04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38"}
Oct 31 19:23:46.786281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:46.786242    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8" event={"ID":"89ca7069-74e3-48c7-9825-37761bdc3b08","Type":"ContainerStarted","Data":"fd9669663b022554e5fa99353993a2294e67caa05b91f2056fac7dc430bd6fce"}
Oct 31 19:23:46.786719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:46.786292    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:23:46.911738 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:46.911685    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8" podStartSLOduration=1.911667647 podStartE2EDuration="1.911667647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:46.905136223 +0000 UTC m=+3730.136374505" watchObservedRunningTime="2026-10-31 19:23:46.911667647 +0000 UTC m=+3730.142905935"
Oct 31 19:23:46.912434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:46.912406    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" path="/var/lib/kubelet/pods/a2c73965-41b1-4e6b-a9e4-f83b798e169d/volumes"
Oct 31 19:23:50.008564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.008528    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"]
Oct 31 19:23:50.009468 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.009439    5551 cpu_manager.go:414] "containerMap: removing container" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy"
Oct 31 19:23:50.009533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.009475    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy"
Oct 31 19:23:50.009639 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.009627    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c73965-41b1-4e6b-a9e4-f83b798e169d" containerName="istio-proxy"
Oct 31 19:23:50.014508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.014486    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.077552 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.077507    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"]
Oct 31 19:23:50.077738 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.077642    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ce8c199032d2\""
Oct 31 19:23:50.078011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.077975    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qmk8p\""
Oct 31 19:23:50.078152 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.078116    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.078325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.078309    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-routing-overlay-volume\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.078389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.078348    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rgf\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-kube-api-access-p9rgf\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.078493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.078451    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-plugins-config-volume\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.179981 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.179942    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-plugins-config-volume\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.180205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.180029    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.180322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.180290    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-routing-overlay-volume\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.180439 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.180363    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rgf\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-kube-api-access-p9rgf\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.180983 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.180952    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-routing-overlay-volume\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.181075 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.180952    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-plugins-config-volume\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.183574 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.183541    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.242183 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.242146    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rgf\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-kube-api-access-p9rgf\") pod \"payload-processing-external-model-686b8c6cf6-56tl9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") " pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.329314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.329227    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.730095 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.730065    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"]
Oct 31 19:23:50.731147 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:23:50.731119    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b2f4f7_7e61_44f9_a974_00de58bb47a9.slice/crio-28a65376b91de63f8cd11e263fc90ee5fb9c88a4f74fa54f389fdf7c4909420e WatchSource:0}: Error finding container 28a65376b91de63f8cd11e263fc90ee5fb9c88a4f74fa54f389fdf7c4909420e: Status 404 returned error can't find the container with id 28a65376b91de63f8cd11e263fc90ee5fb9c88a4f74fa54f389fdf7c4909420e
Oct 31 19:23:50.818965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.818926    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9" event={"ID":"c2b2f4f7-7e61-44f9-a974-00de58bb47a9","Type":"ContainerStarted","Data":"a22d982ff6525fc2ac3928dc8bc97e8b4650a4379e3cf120bfecab30c3fc4a0d"}
Oct 31 19:23:50.819109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.818978    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9" event={"ID":"c2b2f4f7-7e61-44f9-a974-00de58bb47a9","Type":"ContainerStarted","Data":"28a65376b91de63f8cd11e263fc90ee5fb9c88a4f74fa54f389fdf7c4909420e"}
Oct 31 19:23:50.819109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.819009    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:23:50.940356 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:50.940304    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9" podStartSLOduration=1.940289372 podStartE2EDuration="1.940289372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:50.935870811 +0000 UTC m=+3734.167109101" watchObservedRunningTime="2026-10-31 19:23:50.940289372 +0000 UTC m=+3734.171527684"
Oct 31 19:23:51.563936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:51.563894    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"]
Oct 31 19:23:51.564429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:51.564086    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj" podUID="fc13d4d4-721e-4ba8-b7df-ebd258582660" containerName="maas-api" containerID="cri-o://5baf291d6847deca3b3f8dd9bea0c6a4ce6b60465219f22814e016db33bf28e7" gracePeriod=30
Oct 31 19:23:51.828679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:51.828592    5551 generic.go:356] "Generic (PLEG): container finished" podID="fc13d4d4-721e-4ba8-b7df-ebd258582660" containerID="5baf291d6847deca3b3f8dd9bea0c6a4ce6b60465219f22814e016db33bf28e7" exitCode=0
Oct 31 19:23:51.828856 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:51.828669    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj" event={"ID":"fc13d4d4-721e-4ba8-b7df-ebd258582660","Type":"ContainerDied","Data":"5baf291d6847deca3b3f8dd9bea0c6a4ce6b60465219f22814e016db33bf28e7"}
Oct 31 19:23:51.864011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:51.863979    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:23:52.031958 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.031837    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc13d4d4-721e-4ba8-b7df-ebd258582660-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13d4d4-721e-4ba8-b7df-ebd258582660-openshift-service-ca\") pod \"fc13d4d4-721e-4ba8-b7df-ebd258582660\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") "
Oct 31 19:23:52.032359 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.032319    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc13d4d4-721e-4ba8-b7df-ebd258582660-kube-api-access-r2q45\" (UniqueName: \"kubernetes.io/projected/fc13d4d4-721e-4ba8-b7df-ebd258582660-kube-api-access-r2q45\") pod \"fc13d4d4-721e-4ba8-b7df-ebd258582660\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") "
Oct 31 19:23:52.033636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.032575    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-metrics-certs\") pod \"fc13d4d4-721e-4ba8-b7df-ebd258582660\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") "
Oct 31 19:23:52.033636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.032679    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls\") pod \"fc13d4d4-721e-4ba8-b7df-ebd258582660\" (UID: \"fc13d4d4-721e-4ba8-b7df-ebd258582660\") "
Oct 31 19:23:52.038721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.038694    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc13d4d4-721e-4ba8-b7df-ebd258582660-openshift-service-ca" pod "fc13d4d4-721e-4ba8-b7df-ebd258582660" (UID: "fc13d4d4-721e-4ba8-b7df-ebd258582660"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:23:52.050522 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.046373    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-metrics-certs" pod "fc13d4d4-721e-4ba8-b7df-ebd258582660" (UID: "fc13d4d4-721e-4ba8-b7df-ebd258582660"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:23:52.060929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.058866    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls" pod "fc13d4d4-721e-4ba8-b7df-ebd258582660" (UID: "fc13d4d4-721e-4ba8-b7df-ebd258582660"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:23:52.060929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.059999    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc13d4d4-721e-4ba8-b7df-ebd258582660-kube-api-access-r2q45" pod "fc13d4d4-721e-4ba8-b7df-ebd258582660" (UID: "fc13d4d4-721e-4ba8-b7df-ebd258582660"). InnerVolumeSpecName "kube-api-access-r2q45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:52.134136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.134108    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2q45\" (UniqueName: \"kubernetes.io/projected/fc13d4d4-721e-4ba8-b7df-ebd258582660-kube-api-access-r2q45\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:52.134136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.134138    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:52.134322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.134149    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13d4d4-721e-4ba8-b7df-ebd258582660-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:52.134322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.134157    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13d4d4-721e-4ba8-b7df-ebd258582660-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:23:52.837053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.837022    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"
Oct 31 19:23:52.837526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.837056    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj" event={"ID":"fc13d4d4-721e-4ba8-b7df-ebd258582660","Type":"ContainerDied","Data":"7048ccaf6c4adae8d17e39d9c97d90741808b78da479f9fcc9820478adb297db"}
Oct 31 19:23:52.837526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:52.837122    5551 scope.go:122] "RemoveContainer" containerID="5baf291d6847deca3b3f8dd9bea0c6a4ce6b60465219f22814e016db33bf28e7"
Oct 31 19:23:53.007352 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:53.007318    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"]
Oct 31 19:23:53.056553 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:53.056521    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-75c484-cf5d86976-98pnj"]
Oct 31 19:23:54.909499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:54.909448    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc13d4d4-721e-4ba8-b7df-ebd258582660" path="/var/lib/kubelet/pods/fc13d4d4-721e-4ba8-b7df-ebd258582660/volumes"
Oct 31 19:23:57.797358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:23:57.797325    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:24:01.832068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:01.832041    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:24:02.007345 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:02.007313    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"]
Oct 31 19:24:05.534385 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:05.534352    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"]
Oct 31 19:24:05.534824 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:05.534597    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy" containerID="cri-o://b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49" gracePeriod=30
Oct 31 19:24:07.008018 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.007963    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8" podUID="89ca7069-74e3-48c7-9825-37761bdc3b08" containerName="payload-processing" containerID="cri-o://04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38" gracePeriod=25
Oct 31 19:24:07.285370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.285327    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:24:07.339620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.339586    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-provider-credentials\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-provider-credentials\") pod \"89ca7069-74e3-48c7-9825-37761bdc3b08\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") "
Oct 31 19:24:07.339822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.339756    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-routing-overlay-volume\") pod \"89ca7069-74e3-48c7-9825-37761bdc3b08\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") "
Oct 31 19:24:07.339822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.339789    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-kube-api-access-hv5g7\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-kube-api-access-hv5g7\") pod \"89ca7069-74e3-48c7-9825-37761bdc3b08\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") "
Oct 31 19:24:07.339955 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.339833    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-plugins-config-volume\") pod \"89ca7069-74e3-48c7-9825-37761bdc3b08\" (UID: \"89ca7069-74e3-48c7-9825-37761bdc3b08\") "
Oct 31 19:24:07.340196 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.340172    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-routing-overlay-volume" pod "89ca7069-74e3-48c7-9825-37761bdc3b08" (UID: "89ca7069-74e3-48c7-9825-37761bdc3b08"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:07.340361 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.340326    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-plugins-config-volume" pod "89ca7069-74e3-48c7-9825-37761bdc3b08" (UID: "89ca7069-74e3-48c7-9825-37761bdc3b08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:07.340561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.340543    5551 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-routing-overlay-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:07.342003 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.341978    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-provider-credentials" pod "89ca7069-74e3-48c7-9825-37761bdc3b08" (UID: "89ca7069-74e3-48c7-9825-37761bdc3b08"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:07.342098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.342014    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-kube-api-access-hv5g7" pod "89ca7069-74e3-48c7-9825-37761bdc3b08" (UID: "89ca7069-74e3-48c7-9825-37761bdc3b08"). InnerVolumeSpecName "kube-api-access-hv5g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:07.441724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.441682    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5g7\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-kube-api-access-hv5g7\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:07.441724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.441723    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ca7069-74e3-48c7-9825-37761bdc3b08-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:07.442002 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.441740    5551 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89ca7069-74e3-48c7-9825-37761bdc3b08-provider-credentials\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:07.604389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.604307    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:24:07.604389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.604373    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:24:07.947828 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.947796    5551 generic.go:356] "Generic (PLEG): container finished" podID="89ca7069-74e3-48c7-9825-37761bdc3b08" containerID="04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38" exitCode=0
Oct 31 19:24:07.947992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.947844    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"
Oct 31 19:24:07.947992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.947880    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8" event={"ID":"89ca7069-74e3-48c7-9825-37761bdc3b08","Type":"ContainerDied","Data":"04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38"}
Oct 31 19:24:07.947992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.947933    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84869d9799-8gxx8" event={"ID":"89ca7069-74e3-48c7-9825-37761bdc3b08","Type":"ContainerDied","Data":"fd9669663b022554e5fa99353993a2294e67caa05b91f2056fac7dc430bd6fce"}
Oct 31 19:24:07.947992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.947953    5551 scope.go:122] "RemoveContainer" containerID="04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38"
Oct 31 19:24:07.960752 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.960727    5551 scope.go:122] "RemoveContainer" containerID="04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38"
Oct 31 19:24:07.961054 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:07.961034    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38\": container with ID starting with 04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38 not found: ID does not exist" containerID="04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38"
Oct 31 19:24:07.961131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:07.961062    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38"} err="failed to get container status \"04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38\": rpc error: code = NotFound desc = could not find container \"04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38\": container with ID starting with 04f0f7be6aa0a2e7c9047582ba6b9d4ce0bd6b5fb97cbdac8aa384dedfd95b38 not found: ID does not exist"
Oct 31 19:24:08.114012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:08.113976    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"]
Oct 31 19:24:08.120290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:08.120261    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84869d9799-8gxx8"]
Oct 31 19:24:08.910234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:08.910200    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ca7069-74e3-48c7-9825-37761bdc3b08" path="/var/lib/kubelet/pods/89ca7069-74e3-48c7-9825-37761bdc3b08/volumes"
Oct 31 19:24:10.818199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.818110    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:24:10.891287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891251    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-data\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.891544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891310    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-certs\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.891544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891403    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-token\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-token\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.891544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891442    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-socket\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.891544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891496    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-credential-socket\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.891544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891527    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-kube-api-access-m5d6x\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-kube-api-access-m5d6x\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.891830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891642    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-data" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:10.891830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891790    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-envoy\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.891943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891838    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061ccc30-ae82-496c-8a6b-eeec32fd3867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061ccc30-ae82-496c-8a6b-eeec32fd3867-istiod-ca-cert\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.891943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891864    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-credential-socket" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:10.891943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891864    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-socket" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:10.891943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.891936    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-podinfo\") pod \"061ccc30-ae82-496c-8a6b-eeec32fd3867\" (UID: \"061ccc30-ae82-496c-8a6b-eeec32fd3867\") "
Oct 31 19:24:10.892225 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.892201    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-certs" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:10.892322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.892307    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061ccc30-ae82-496c-8a6b-eeec32fd3867-istiod-ca-cert" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:10.892693 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.892666    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:10.892799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.892698    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:10.892799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.892714    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061ccc30-ae82-496c-8a6b-eeec32fd3867-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:10.892799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.892731    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:10.892799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.892746    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:10.894845 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.894813    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-kube-api-access-m5d6x" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "kube-api-access-m5d6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:10.894976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.894870    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-token" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:10.895065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.895045    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-envoy" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:10.895265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.895235    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-podinfo" pod "061ccc30-ae82-496c-8a6b-eeec32fd3867" (UID: "061ccc30-ae82-496c-8a6b-eeec32fd3867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:24:10.975062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.975030    5551 generic.go:356] "Generic (PLEG): container finished" podID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerID="b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49" exitCode=0
Oct 31 19:24:10.975319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.975085    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" event={"ID":"061ccc30-ae82-496c-8a6b-eeec32fd3867","Type":"ContainerDied","Data":"b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49"}
Oct 31 19:24:10.975319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.975105    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"
Oct 31 19:24:10.975319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.975136    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v" event={"ID":"061ccc30-ae82-496c-8a6b-eeec32fd3867","Type":"ContainerDied","Data":"8a32c0c750977a860f3774a0b74dd5962996d1fc4cc549f10901f275c7750cd4"}
Oct 31 19:24:10.975319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.975161    5551 scope.go:122] "RemoveContainer" containerID="b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49"
Oct 31 19:24:10.989512 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.989483    5551 scope.go:122] "RemoveContainer" containerID="756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450"
Oct 31 19:24:10.994697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.994670    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:10.994817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.994706    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5d6x\" (UniqueName: \"kubernetes.io/projected/061ccc30-ae82-496c-8a6b-eeec32fd3867-kube-api-access-m5d6x\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:10.994817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.994724    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:10.994817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:10.994738    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061ccc30-ae82-496c-8a6b-eeec32fd3867-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:11.003630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:11.003613    5551 scope.go:122] "RemoveContainer" containerID="b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49"
Oct 31 19:24:11.004180 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:11.004035    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49\": container with ID starting with b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49 not found: ID does not exist" containerID="b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49"
Oct 31 19:24:11.004180 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:11.004070    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49"} err="failed to get container status \"b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49\": rpc error: code = NotFound desc = could not find container \"b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49\": container with ID starting with b997e08f8123e6ef598ac282dc6ab0991a59d1c212ecab873fcf327430bf9a49 not found: ID does not exist"
Oct 31 19:24:11.004180 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:11.004095    5551 scope.go:122] "RemoveContainer" containerID="756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450"
Oct 31 19:24:11.008320 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:11.004742    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450\": container with ID starting with 756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450 not found: ID does not exist" containerID="756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450"
Oct 31 19:24:11.008542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:11.004776    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450"} err="failed to get container status \"756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450\": rpc error: code = NotFound desc = could not find container \"756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450\": container with ID starting with 756cab65bdd6df73dbaf0dfd1b53b244d85bd4c5a8a4797024f2314ea2909450 not found: ID does not exist"
Oct 31 19:24:11.114979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:11.114903    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"]
Oct 31 19:24:11.121429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:11.121392    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-75c484-openshift-default-77f4f44d9c-ftz8v"]
Oct 31 19:24:12.909866 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:12.909826    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" path="/var/lib/kubelet/pods/061ccc30-ae82-496c-8a6b-eeec32fd3867/volumes"
Oct 31 19:24:16.604408 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:16.604362    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:24:25.420263 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.420230    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"]
Oct 31 19:24:25.420860 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.420843    5551 cpu_manager.go:414] "containerMap: removing container" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy"
Oct 31 19:24:25.420905 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.420861    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy"
Oct 31 19:24:25.420905 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.420874    5551 cpu_manager.go:414] "containerMap: removing container" podUID="fc13d4d4-721e-4ba8-b7df-ebd258582660" containerName="maas-api"
Oct 31 19:24:25.420905 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.420879    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc13d4d4-721e-4ba8-b7df-ebd258582660" containerName="maas-api"
Oct 31 19:24:25.420905 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.420896    5551 cpu_manager.go:414] "containerMap: removing container" podUID="89ca7069-74e3-48c7-9825-37761bdc3b08" containerName="payload-processing"
Oct 31 19:24:25.420905 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.420900    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ca7069-74e3-48c7-9825-37761bdc3b08" containerName="payload-processing"
Oct 31 19:24:25.421070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.421005    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy"
Oct 31 19:24:25.421070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.421014    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc13d4d4-721e-4ba8-b7df-ebd258582660" containerName="maas-api"
Oct 31 19:24:25.421070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.421021    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ca7069-74e3-48c7-9825-37761bdc3b08" containerName="payload-processing"
Oct 31 19:24:25.426193 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.426173    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.486357 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.486331    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nsw28\""
Oct 31 19:24:25.543840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.543812    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"]
Oct 31 19:24:25.580348 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.580314    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn6xh\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-kube-api-access-qn6xh\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.580528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.580363    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05089d3e-fe0e-47a2-b352-1625eb25fd09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.580528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.580489    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.580645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.580584    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.580688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.580647    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.580742 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.580704    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.580842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.580823    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.580945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.580932    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.581032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.581014    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682228    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682327 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682304    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682356    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682414    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn6xh\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-kube-api-access-qn6xh\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682444    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05089d3e-fe0e-47a2-b352-1625eb25fd09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682501    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682540    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682601    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.682695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682662    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.683102 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682731    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.683102 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.682780    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.683226 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.683184    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.683297 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.683257    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.683494 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.683442    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05089d3e-fe0e-47a2-b352-1625eb25fd09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.685206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.685186    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.685682 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.685661    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.739394 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.739364    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:25.739692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:25.739669    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn6xh\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-kube-api-access-qn6xh\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bkh82\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:26.040574 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:26.040483    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:26.484301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:26.484273    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"]
Oct 31 19:24:26.487180 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:24:26.487146    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05089d3e_fe0e_47a2_b352_1625eb25fd09.slice/crio-9e059f6639332a0d247610114fd59095a6dbd72a9b71e962fed1dc5ee1e4c476 WatchSource:0}: Error finding container 9e059f6639332a0d247610114fd59095a6dbd72a9b71e962fed1dc5ee1e4c476: Status 404 returned error can't find the container with id 9e059f6639332a0d247610114fd59095a6dbd72a9b71e962fed1dc5ee1e4c476
Oct 31 19:24:26.490266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:26.490229    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:24:26.490385 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:26.490305    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:24:26.490385 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:26.490337    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:24:27.105524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:27.105481    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" event={"ID":"05089d3e-fe0e-47a2-b352-1625eb25fd09","Type":"ContainerStarted","Data":"242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724"}
Oct 31 19:24:27.105742 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:27.105554    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" event={"ID":"05089d3e-fe0e-47a2-b352-1625eb25fd09","Type":"ContainerStarted","Data":"9e059f6639332a0d247610114fd59095a6dbd72a9b71e962fed1dc5ee1e4c476"}
Oct 31 19:24:27.323131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:27.323066    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podStartSLOduration=2.323050942 podStartE2EDuration="2.323050942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:27.271145961 +0000 UTC m=+3770.502384252" watchObservedRunningTime="2026-10-31 19:24:27.323050942 +0000 UTC m=+3770.554289231"
Oct 31 19:24:28.042444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:28.042405    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:28.043585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:28.043562    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:24:28.043689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:28.043611    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 31 19:24:29.041210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:29.041173    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:24:29.041392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:29.041223    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 31 19:24:30.040901 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:30.040869    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:24:30.041309 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:30.040936    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 31 19:24:31.041958 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:31.041917    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:24:31.041958 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:31.041971    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 31 19:24:32.041567 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:32.041534    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:24:32.041741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:32.041587    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 31 19:24:33.040954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.040909    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:24:33.041515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.040969    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 31 19:24:33.321397 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.321308    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"]
Oct 31 19:24:33.322245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.322217    5551 cpu_manager.go:414] "containerMap: removing container" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy"
Oct 31 19:24:33.322245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.322246    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy"
Oct 31 19:24:33.322957 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.322388    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="061ccc30-ae82-496c-8a6b-eeec32fd3867" containerName="istio-proxy"
Oct 31 19:24:33.326245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.326226    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.403301 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.403264    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6wc\" (UniqueName: \"kubernetes.io/projected/2cf3395c-b065-43e2-af60-9ce6ee366441-kube-api-access-kn6wc\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.403534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.403509    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-metrics-certs\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.403605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.403574    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.403671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.403638    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3395c-b065-43e2-af60-9ce6ee366441-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.428473 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.428428    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-b70248-serving-cert\""
Oct 31 19:24:33.433741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.433704    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"]
Oct 31 19:24:33.505438 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.505395    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-metrics-certs\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.505615 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.505479    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.505615 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.505501    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3395c-b065-43e2-af60-9ce6ee366441-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.505615 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.505604    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6wc\" (UniqueName: \"kubernetes.io/projected/2cf3395c-b065-43e2-af60-9ce6ee366441-kube-api-access-kn6wc\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.505749 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:33.505619    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-serving-cert: secret "maas-api-e2e-extmod-a-b70248-serving-cert" not found
Oct 31 19:24:33.505749 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:33.505693    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls podName:2cf3395c-b065-43e2-af60-9ce6ee366441 nodeName:}" failed. No retries permitted until 2026-10-31 19:24:34.005672554 +0000 UTC m=+3777.236910843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls") pod "maas-api-e2e-extmod-a-b70248-778c84f566-vh86s" (UID: "2cf3395c-b065-43e2-af60-9ce6ee366441") : secret "maas-api-e2e-extmod-a-b70248-serving-cert" not found
Oct 31 19:24:33.506250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.506179    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3395c-b065-43e2-af60-9ce6ee366441-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.508047 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.508024    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-metrics-certs\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:33.566561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:33.566531    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6wc\" (UniqueName: \"kubernetes.io/projected/2cf3395c-b065-43e2-af60-9ce6ee366441-kube-api-access-kn6wc\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:34.013231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:34.013190    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:34.016028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:34.015992    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b70248-778c84f566-vh86s\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:34.042480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:34.041749    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:24:34.042480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:34.041804    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 31 19:24:34.242271 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:34.242241    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:34.630760 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:34.630734    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"]
Oct 31 19:24:34.633076 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:24:34.633044    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf3395c_b065_43e2_af60_9ce6ee366441.slice/crio-ad00ea2b206f3a16a1dbcd210d954999ff5d31d98efe7e11423532d606b609fd WatchSource:0}: Error finding container ad00ea2b206f3a16a1dbcd210d954999ff5d31d98efe7e11423532d606b609fd: Status 404 returned error can't find the container with id ad00ea2b206f3a16a1dbcd210d954999ff5d31d98efe7e11423532d606b609fd
Oct 31 19:24:34.634926 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:34.634906    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:24:35.167420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:35.167380    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s" event={"ID":"2cf3395c-b065-43e2-af60-9ce6ee366441","Type":"ContainerStarted","Data":"ad00ea2b206f3a16a1dbcd210d954999ff5d31d98efe7e11423532d606b609fd"}
Oct 31 19:24:36.041988 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:36.041953    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:24:36.042247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:36.042216    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:24:36.178974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:36.178926    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s" event={"ID":"2cf3395c-b065-43e2-af60-9ce6ee366441","Type":"ContainerStarted","Data":"247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7"}
Oct 31 19:24:36.179360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:36.179091    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:36.296828 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:36.296733    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s" podStartSLOduration=2.428071173 podStartE2EDuration="3.296718666s" totalImagesPullingTime="868.647493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:36.293877375 +0000 UTC m=+3779.525115668" watchObservedRunningTime="2026-10-31 19:24:36.296718666 +0000 UTC m=+3779.527956955"
Oct 31 19:24:37.084625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:37.084581    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bkh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:24:37.084839 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:37.084650    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:24:38.016773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:38.016740    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:38.017604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:38.017573    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:38.020055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:38.020028    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:24:42.192551 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:42.192511    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:24:44.872686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:44.872656    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"]
Oct 31 19:24:44.873155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:44.872836    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy" containerID="cri-o://c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5" gracePeriod=30
Oct 31 19:24:44.876953 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:44.876930    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"]
Oct 31 19:24:44.877137 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:44.877117    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy" containerID="cri-o://cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667" gracePeriod=30
Oct 31 19:24:46.606082 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:46.606043    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:24:48.809425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:48.809385    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:24:48.809903 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:48.809467    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:24:49.000707 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:49.000670    5551 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:24:49.000997 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:49.000963    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:24:50.176859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.176831    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:24:50.205732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.205705    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-socket\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.205871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.205785    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-data\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.205871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.205834    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-credential-socket\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.205871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.205861    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-certs\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.206044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.205953    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-token\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-token\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.206044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.205986    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istiod-ca-cert\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.206044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.205976    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-socket" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.206044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206037    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-data" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.206265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206066    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-envoy\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.206265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206128    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-kube-api-access-dtwwh\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-kube-api-access-dtwwh\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.206265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206132    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-credential-socket" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.206265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206167    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-podinfo\") pod \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\" (UID: \"c020660c-8b71-48a0-a2e8-7b09c6e253cb\") "
Oct 31 19:24:50.206506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206319    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istiod-ca-cert" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:50.206857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206807    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.206857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206836    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.206857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206851    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.206857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206865    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.206857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.206811    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-certs" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.213888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.208435    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-podinfo" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:24:50.213888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.208573    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-token" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:50.213888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.208917    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-envoy" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.213888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.209193    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-kube-api-access-dtwwh" pod "c020660c-8b71-48a0-a2e8-7b09c6e253cb" (UID: "c020660c-8b71-48a0-a2e8-7b09c6e253cb"). InnerVolumeSpecName "kube-api-access-dtwwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:50.222436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.222406    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:24:50.307505 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307449    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74636d56-5323-4d54-9251-35dcf230256f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74636d56-5323-4d54-9251-35dcf230256f-istiod-ca-cert\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.307700 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307536    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-socket\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.307700 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307571    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-kube-api-access-vwntr\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-kube-api-access-vwntr\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.307700 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307596    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74636d56-5323-4d54-9251-35dcf230256f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74636d56-5323-4d54-9251-35dcf230256f-istio-podinfo\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.307700 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307689    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-istio-token\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-istio-token\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.307935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307797    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-credential-socket\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.307935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307798    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-socket" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.307935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307835    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-data\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.307935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307866    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-certs\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.307935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307867    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74636d56-5323-4d54-9251-35dcf230256f-istiod-ca-cert" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:50.307935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.307914    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-envoy\") pod \"74636d56-5323-4d54-9251-35dcf230256f\" (UID: \"74636d56-5323-4d54-9251-35dcf230256f\") "
Oct 31 19:24:50.308395 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.308375    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtwwh\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-kube-api-access-dtwwh\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.308807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.308403    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.308807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.308418    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.308807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.308431    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.308807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.308445    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74636d56-5323-4d54-9251-35dcf230256f-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.308807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.308477    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.308807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.308493    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c020660c-8b71-48a0-a2e8-7b09c6e253cb-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.309201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.308942    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-data" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.309201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.309139    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-certs" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.309322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.309278    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-credential-socket" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.309971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.309941    5551 generic.go:356] "Generic (PLEG): container finished" podID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerID="cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667" exitCode=0
Oct 31 19:24:50.310094 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.310071    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" event={"ID":"c020660c-8b71-48a0-a2e8-7b09c6e253cb","Type":"ContainerDied","Data":"cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667"}
Oct 31 19:24:50.310164 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.310138    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6" event={"ID":"c020660c-8b71-48a0-a2e8-7b09c6e253cb","Type":"ContainerDied","Data":"f86983c6a7763aed488a4c38ce4f49670192906d782c6479662f4f0f4b656b6f"}
Oct 31 19:24:50.310223 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.310172    5551 scope.go:122] "RemoveContainer" containerID="cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667"
Oct 31 19:24:50.310366 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.310347    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"
Oct 31 19:24:50.310590 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.310447    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-istio-token" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:50.310590 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.310514    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-kube-api-access-vwntr" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "kube-api-access-vwntr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:50.310942 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.310858    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-envoy" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.312175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.312151    5551 generic.go:356] "Generic (PLEG): container finished" podID="74636d56-5323-4d54-9251-35dcf230256f" containerID="c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5" exitCode=0
Oct 31 19:24:50.312278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.312209    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"
Oct 31 19:24:50.312278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.312220    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" event={"ID":"74636d56-5323-4d54-9251-35dcf230256f","Type":"ContainerDied","Data":"c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5"}
Oct 31 19:24:50.312278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.312260    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw" event={"ID":"74636d56-5323-4d54-9251-35dcf230256f","Type":"ContainerDied","Data":"2a43b1421c2652f12b9507d24fe7b821137b0f205a6733f0a8028160acdfb46b"}
Oct 31 19:24:50.313599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.313576    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74636d56-5323-4d54-9251-35dcf230256f-istio-podinfo" pod "74636d56-5323-4d54-9251-35dcf230256f" (UID: "74636d56-5323-4d54-9251-35dcf230256f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:24:50.322602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.322582    5551 scope.go:122] "RemoveContainer" containerID="693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63"
Oct 31 19:24:50.337762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.337601    5551 scope.go:122] "RemoveContainer" containerID="cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667"
Oct 31 19:24:50.339839 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:50.339813    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667\": container with ID starting with cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667 not found: ID does not exist" containerID="cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667"
Oct 31 19:24:50.339954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.339847    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667"} err="failed to get container status \"cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667\": rpc error: code = NotFound desc = could not find container \"cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667\": container with ID starting with cbd0fd913430c59bf77654030da33cf01f1b9cc8b616558a8509caf163258667 not found: ID does not exist"
Oct 31 19:24:50.339954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.339866    5551 scope.go:122] "RemoveContainer" containerID="693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63"
Oct 31 19:24:50.340209 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:50.340184    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63\": container with ID starting with 693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63 not found: ID does not exist" containerID="693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63"
Oct 31 19:24:50.340380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.340318    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63"} err="failed to get container status \"693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63\": rpc error: code = NotFound desc = could not find container \"693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63\": container with ID starting with 693eec0e01bcd0dc1d249f5a0f26a8c749e329c664bc24ba6dc4a3c74a218a63 not found: ID does not exist"
Oct 31 19:24:50.340380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.340353    5551 scope.go:122] "RemoveContainer" containerID="c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5"
Oct 31 19:24:50.352805 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.352773    5551 scope.go:122] "RemoveContainer" containerID="c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5"
Oct 31 19:24:50.353118 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:24:50.353095    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5\": container with ID starting with c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5 not found: ID does not exist" containerID="c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5"
Oct 31 19:24:50.353204 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.353126    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5"} err="failed to get container status \"c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5\": rpc error: code = NotFound desc = could not find container \"c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5\": container with ID starting with c4b5d21739545313621bc890b5620a29aa46a1af0a333aeb58d14eba8f3cc3f5 not found: ID does not exist"
Oct 31 19:24:50.409274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.409209    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwntr\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-kube-api-access-vwntr\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.409274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.409236    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74636d56-5323-4d54-9251-35dcf230256f-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.409274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.409253    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74636d56-5323-4d54-9251-35dcf230256f-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.409274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.409266    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.409541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.409280    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.409541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.409293    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.409541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.409304    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74636d56-5323-4d54-9251-35dcf230256f-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.600783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.600745    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"]
Oct 31 19:24:50.625844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.625818    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-xj8z6"]
Oct 31 19:24:50.735581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.735487    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"]
Oct 31 19:24:50.738482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.738443    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5jbnw"]
Oct 31 19:24:50.911254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.911219    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74636d56-5323-4d54-9251-35dcf230256f" path="/var/lib/kubelet/pods/74636d56-5323-4d54-9251-35dcf230256f/volumes"
Oct 31 19:24:50.911898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:50.911876    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" path="/var/lib/kubelet/pods/c020660c-8b71-48a0-a2e8-7b09c6e253cb/volumes"
Oct 31 19:24:58.756313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.756276    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"]
Oct 31 19:24:58.756867 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.756853    5551 cpu_manager.go:414] "containerMap: removing container" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy"
Oct 31 19:24:58.756912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.756869    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy"
Oct 31 19:24:58.756912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.756897    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy"
Oct 31 19:24:58.756912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.756903    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy"
Oct 31 19:24:58.757011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.757006    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy"
Oct 31 19:24:58.757049 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.757021    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="74636d56-5323-4d54-9251-35dcf230256f" containerName="istio-proxy"
Oct 31 19:24:58.761997 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.761977    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.815971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.815938    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c97ce8-openshift-default-dockercfg-6mwl6\""
Oct 31 19:24:58.864842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.863858    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-token\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.865379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.865341    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8696\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-kube-api-access-f8696\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.866410 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.866377    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-podinfo\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.866605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.866441    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-socket\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.866605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.866508    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-certs\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.867231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.866650    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-envoy\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.867231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.866737    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-credential-socket\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.867231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.866790    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istiod-ca-cert\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.867231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.866881    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-data\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.870883 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.870586    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"]
Oct 31 19:24:58.968098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968008    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-podinfo\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968072    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-socket\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968140    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-certs\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968178    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-envoy\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968220    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-credential-socket\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968265    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istiod-ca-cert\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968327    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-data\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968484    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-token\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968533    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8696\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-kube-api-access-f8696\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968670    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-socket\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.968969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968728    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-credential-socket\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.969087 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.968970    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-certs\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.969237 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.969205    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-data\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.969400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.969377    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istiod-ca-cert\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.974499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.972302    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-podinfo\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:58.974499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:58.972560    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-envoy\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:59.025925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:59.025859    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-token\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:59.026315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:59.026295    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8696\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-kube-api-access-f8696\") pod \"e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") " pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:59.077755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:59.077725    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:24:59.531591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:59.531564    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"]
Oct 31 19:24:59.532350 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:24:59.532317    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa859b3_2ed2_4a73_bfa6_51028ab181b1.slice/crio-559e926fe7ae22b76db3538489d15f5e42e7c4c78a93d19fe05d9332e82988ee WatchSource:0}: Error finding container 559e926fe7ae22b76db3538489d15f5e42e7c4c78a93d19fe05d9332e82988ee: Status 404 returned error can't find the container with id 559e926fe7ae22b76db3538489d15f5e42e7c4c78a93d19fe05d9332e82988ee
Oct 31 19:24:59.534867 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:59.534830    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:24:59.534969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:59.534909    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:24:59.534969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:24:59.534957    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:00.392789 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:00.392745    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" event={"ID":"8fa859b3-2ed2-4a73-bfa6-51028ab181b1","Type":"ContainerStarted","Data":"21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c"}
Oct 31 19:25:00.393194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:00.392832    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" event={"ID":"8fa859b3-2ed2-4a73-bfa6-51028ab181b1","Type":"ContainerStarted","Data":"559e926fe7ae22b76db3538489d15f5e42e7c4c78a93d19fe05d9332e82988ee"}
Oct 31 19:25:00.629482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:00.624447    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" podStartSLOduration=2.6244221 podStartE2EDuration="2.6244221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:00.593946765 +0000 UTC m=+3803.825185100" watchObservedRunningTime="2026-10-31 19:25:00.6244221 +0000 UTC m=+3803.855660389"
Oct 31 19:25:01.078099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:01.078067    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:25:01.083871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:01.083848    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:25:01.407483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:01.406440    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:25:01.409517 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:01.408740    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:25:10.672298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.672267    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"]
Oct 31 19:25:10.672920 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.672905    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy"
Oct 31 19:25:10.672974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.672921    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy"
Oct 31 19:25:10.673018 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.673003    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c020660c-8b71-48a0-a2e8-7b09c6e253cb" containerName="istio-proxy"
Oct 31 19:25:10.676601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.676583    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.737840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.737802    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"]
Oct 31 19:25:10.772636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.772596    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.772826 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.772650    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.772826 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.772723    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.772826 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.772770    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b0a99b-0d21-4232-a701-7f14c6d33748-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.773043 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.772826    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gx5\" (UniqueName: \"kubernetes.io/projected/a7b0a99b-0d21-4232-a701-7f14c6d33748-kube-api-access-72gx5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.773043 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.772937    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.773043 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.773001    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.773239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.773047    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.773239 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.773146    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.874720 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.874677    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.874903 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.874809    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.874903 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.874893    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.874928    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.874986    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.875032    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b0a99b-0d21-4232-a701-7f14c6d33748-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875237 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.875088    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gx5\" (UniqueName: \"kubernetes.io/projected/a7b0a99b-0d21-4232-a701-7f14c6d33748-kube-api-access-72gx5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875237 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.875162    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875352 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.875234    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875352 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.875259    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.875431    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875629 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.875595    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.875898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.875872    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b0a99b-0d21-4232-a701-7f14c6d33748-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.876160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.876025    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.878141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.878111    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.878710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.878689    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.932575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.932444    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gx5\" (UniqueName: \"kubernetes.io/projected/a7b0a99b-0d21-4232-a701-7f14c6d33748-kube-api-access-72gx5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.932747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.932700    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b0a99b-0d21-4232-a701-7f14c6d33748-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-p9pnj\" (UID: \"a7b0a99b-0d21-4232-a701-7f14c6d33748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:10.995044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:10.995014    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:11.438067 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:11.438030    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"]
Oct 31 19:25:11.489916 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:25:11.489884    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b0a99b_0d21_4232_a701_7f14c6d33748.slice/crio-1aa77b8b06e74c48151686815acf5fedc68b179fbe33b9a01d656c3731805366 WatchSource:0}: Error finding container 1aa77b8b06e74c48151686815acf5fedc68b179fbe33b9a01d656c3731805366: Status 404 returned error can't find the container with id 1aa77b8b06e74c48151686815acf5fedc68b179fbe33b9a01d656c3731805366
Oct 31 19:25:11.492872 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:11.492828    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:11.492979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:11.492926    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:11.492979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:11.492959    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:12.521788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:12.521746    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" event={"ID":"a7b0a99b-0d21-4232-a701-7f14c6d33748","Type":"ContainerStarted","Data":"e961f2ce950de3714f915793204a783f6716cd3cf4b5398a3b1b644ed9d44aa0"}
Oct 31 19:25:12.522274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:12.521818    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" event={"ID":"a7b0a99b-0d21-4232-a701-7f14c6d33748","Type":"ContainerStarted","Data":"1aa77b8b06e74c48151686815acf5fedc68b179fbe33b9a01d656c3731805366"}
Oct 31 19:25:12.738883 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:12.738792    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podStartSLOduration=2.738771915 podStartE2EDuration="2.738771915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:12.68786104 +0000 UTC m=+3815.919099330" watchObservedRunningTime="2026-10-31 19:25:12.738771915 +0000 UTC m=+3815.970010206"
Oct 31 19:25:12.995387 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:12.995351    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:12.997240 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:12.997204    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 31 19:25:12.997394 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:12.997260    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 31 19:25:13.996177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:13.996141    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 31 19:25:13.996636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:13.996199    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 31 19:25:14.995599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:14.995564    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 31 19:25:14.995822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:14.995616    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 31 19:25:15.680344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.680304    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"]
Oct 31 19:25:15.684823 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.684801    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.737000 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.736968    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fc050d9d3c08\""
Oct 31 19:25:15.786415 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.786370    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9xp8\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-kube-api-access-z9xp8\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.786764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.786727    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.786899 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.786796    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-routing-overlay-volume\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.786970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.786897    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-plugins-config-volume\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.791492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.791449    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"]
Oct 31 19:25:15.888245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.888208    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9xp8\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-kube-api-access-z9xp8\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.888435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.888255    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.888435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.888283    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-routing-overlay-volume\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.888435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.888337    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-plugins-config-volume\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.889043 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.889017    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-routing-overlay-volume\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.889138 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.889077    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-plugins-config-volume\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.890956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.890935    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.946343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.946255    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9xp8\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-kube-api-access-z9xp8\") pod \"payload-processing-external-model-dfd4695f9-ggtv6\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") " pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:15.995518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.995451    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 31 19:25:15.995717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:15.995540    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 31 19:25:16.001475 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:16.001437    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:16.415530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:16.415502    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"]
Oct 31 19:25:16.417436 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:25:16.417399    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34fe25e_eb40_40eb_ab9b_c5240dfd6b29.slice/crio-cb0a12b60c6420faa849d18b450fe6979bb6aea2f38f763eba0b3b0dd59e6894 WatchSource:0}: Error finding container cb0a12b60c6420faa849d18b450fe6979bb6aea2f38f763eba0b3b0dd59e6894: Status 404 returned error can't find the container with id cb0a12b60c6420faa849d18b450fe6979bb6aea2f38f763eba0b3b0dd59e6894
Oct 31 19:25:16.576425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:16.575316    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6" event={"ID":"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29","Type":"ContainerStarted","Data":"7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747"}
Oct 31 19:25:16.576425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:16.575381    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6" event={"ID":"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29","Type":"ContainerStarted","Data":"cb0a12b60c6420faa849d18b450fe6979bb6aea2f38f763eba0b3b0dd59e6894"}
Oct 31 19:25:16.576425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:16.576374    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:16.607730 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:16.607695    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:25:16.689482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:16.689349    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6" podStartSLOduration=1.689329989 podStartE2EDuration="1.689329989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:16.68787906 +0000 UTC m=+3819.919117363" watchObservedRunningTime="2026-10-31 19:25:16.689329989 +0000 UTC m=+3819.920568294"
Oct 31 19:25:16.995320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:16.995242    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 31 19:25:16.995320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:16.995292    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 31 19:25:17.995390 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:17.995357    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 31 19:25:17.995872 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:17.995415    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 31 19:25:18.729059 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:18.729030    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fc050d9d3c08: secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:18.729318 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:18.729301    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6: secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:18.729535 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:18.729520    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials podName:b34fe25e-eb40-40eb-ab9b-c5240dfd6b29 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:19.229495086 +0000 UTC m=+3822.460733356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials") pod "payload-processing-external-model-dfd4695f9-ggtv6" (UID: "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29") : secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:18.995360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:18.995268    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 31 19:25:18.995360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:18.995332    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 31 19:25:19.240747 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:19.240716    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fc050d9d3c08: secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:19.240747 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:19.240742    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6: secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:19.241024 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:19.240803    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials podName:b34fe25e-eb40-40eb-ab9b-c5240dfd6b29 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:20.240786603 +0000 UTC m=+3823.472024891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials") pod "payload-processing-external-model-dfd4695f9-ggtv6" (UID: "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29") : secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:19.950618 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:19.950560    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:19.950618 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:19.950623    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:19.950928 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:19.950693    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:20.450670332 +0000 UTC m=+3823.681908620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:20.261381 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:20.261298    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fc050d9d3c08: secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:20.261381 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:20.261330    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6: secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:20.261828 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:20.261402    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials podName:b34fe25e-eb40-40eb-ab9b-c5240dfd6b29 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:22.261379662 +0000 UTC m=+3825.492617948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials") pod "payload-processing-external-model-dfd4695f9-ggtv6" (UID: "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29") : secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:20.370536 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:20.370498    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:20.371079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:20.371050    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:20.371951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:20.371928    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj"
Oct 31 19:25:20.464137 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:20.464104    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:20.464137 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:20.464140    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:20.464395 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:20.464208    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:21.464188411 +0000 UTC m=+3824.695426678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:20.858714 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:20.858672    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"]
Oct 31 19:25:21.084369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.084329    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"]
Oct 31 19:25:21.090924 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.090884    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.194348 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.194317    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ad2d15-openshift-default-dockercfg-dpfmc\""
Oct 31 19:25:21.208488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.204950    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-socket\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.223564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.223053    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"]
Oct 31 19:25:21.314740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.314678    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-data\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.315371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.314770    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-envoy\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.315371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.314804    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmmn\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-kube-api-access-9kmmn\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.315371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.314852    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-socket\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.315371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.314922    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istiod-ca-cert\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.315371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.315058    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-credential-socket\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.315371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.315148    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-certs\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.315371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.315186    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-podinfo\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.315371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.315227    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-token\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.316019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.315900    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-socket\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.416659 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.416619    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-data\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.416881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.416692    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-envoy\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.416881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.416821    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmmn\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-kube-api-access-9kmmn\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.416930    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istiod-ca-cert\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.417024    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-credential-socket\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.417040    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-data\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.417126    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-certs\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.417167    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-podinfo\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.417214    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-token\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417359 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.417343    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-credential-socket\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417665 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.417629    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-certs\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.417885 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.417861    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istiod-ca-cert\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.419349 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.419328    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-envoy\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.419918 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.419898    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-podinfo\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.474914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.474831    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-token\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.475475 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.475434    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmmn\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-kube-api-access-9kmmn\") pod \"e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") " pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:21.518473 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:21.518421    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:21.518473 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:21.518474    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:21.518715 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:21.518658    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:23.518632726 +0000 UTC m=+3826.749871014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:21.707864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:21.707830    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:22.145390 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:22.145361    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"]
Oct 31 19:25:22.146601 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:25:22.146573    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9ac8e8_f19e_4ccb_a5f5_e08cfb3c10b4.slice/crio-bb1998fdf41aae6e5138404d481865fcf2f9ae551f731dfda5991d4a262a7d67 WatchSource:0}: Error finding container bb1998fdf41aae6e5138404d481865fcf2f9ae551f731dfda5991d4a262a7d67: Status 404 returned error can't find the container with id bb1998fdf41aae6e5138404d481865fcf2f9ae551f731dfda5991d4a262a7d67
Oct 31 19:25:22.149591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:22.149554    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:22.149694 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:22.149625    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:22.149694 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:22.149661    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:22.330270 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:22.330227    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fc050d9d3c08: secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:22.330270 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:22.330256    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6: secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:22.330830 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:22.330311    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials podName:b34fe25e-eb40-40eb-ab9b-c5240dfd6b29 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:26.330296441 +0000 UTC m=+3829.561534739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials") pod "payload-processing-external-model-dfd4695f9-ggtv6" (UID: "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29") : secret "agc-cred-fc050d9d3c08" not found
Oct 31 19:25:22.629123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:22.629082    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" event={"ID":"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4","Type":"ContainerStarted","Data":"f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd"}
Oct 31 19:25:22.629313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:22.629153    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" event={"ID":"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4","Type":"ContainerStarted","Data":"bb1998fdf41aae6e5138404d481865fcf2f9ae551f731dfda5991d4a262a7d67"}
Oct 31 19:25:22.708498 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:22.708444    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:22.752569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:22.752512    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podStartSLOduration=1.752496756 podStartE2EDuration="1.752496756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:22.748742749 +0000 UTC m=+3825.979981050" watchObservedRunningTime="2026-10-31 19:25:22.752496756 +0000 UTC m=+3825.983735047"
Oct 31 19:25:23.547966 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:23.547928    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:23.547966 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:23.547962    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:23.548586 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:23.548020    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:27.548003904 +0000 UTC m=+3830.779242188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:23.714954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:23.714916    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:24.647288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:24.647255    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:24.648337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:24.648313    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:25:25.859248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:25.859201    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6" podUID="b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" containerName="payload-processing" containerID="cri-o://7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747" gracePeriod=25
Oct 31 19:25:26.128671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.128648    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:26.230159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.230124    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-kube-api-access-z9xp8\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-kube-api-access-z9xp8\") pod \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") "
Oct 31 19:25:26.230322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.230199    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-routing-overlay-volume\") pod \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") "
Oct 31 19:25:26.230322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.230244    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials\") pod \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") "
Oct 31 19:25:26.230486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.230384    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-plugins-config-volume\") pod \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\" (UID: \"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29\") "
Oct 31 19:25:26.230671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.230636    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-routing-overlay-volume" pod "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" (UID: "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:25:26.230887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.230857    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-plugins-config-volume" pod "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" (UID: "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:25:26.231091 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.231073    5551 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-routing-overlay-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:26.231151 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.231098    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:26.232980 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.232958    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials" pod "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" (UID: "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:26.233057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.233008    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-kube-api-access-z9xp8" pod "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" (UID: "b34fe25e-eb40-40eb-ab9b-c5240dfd6b29"). InnerVolumeSpecName "kube-api-access-z9xp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:26.332310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.332279    5551 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-provider-credentials\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:26.332310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.332304    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9xp8\" (UniqueName: \"kubernetes.io/projected/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29-kube-api-access-z9xp8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:26.664432 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.664396    5551 generic.go:356] "Generic (PLEG): container finished" podID="b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" containerID="7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747" exitCode=0
Oct 31 19:25:26.664662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.664477    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"
Oct 31 19:25:26.664662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.664506    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6" event={"ID":"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29","Type":"ContainerDied","Data":"7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747"}
Oct 31 19:25:26.664662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.664576    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6" event={"ID":"b34fe25e-eb40-40eb-ab9b-c5240dfd6b29","Type":"ContainerDied","Data":"cb0a12b60c6420faa849d18b450fe6979bb6aea2f38f763eba0b3b0dd59e6894"}
Oct 31 19:25:26.664662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.664609    5551 scope.go:122] "RemoveContainer" containerID="7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747"
Oct 31 19:25:26.676719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.676697    5551 scope.go:122] "RemoveContainer" containerID="7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747"
Oct 31 19:25:26.677064 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:26.677039    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747\": container with ID starting with 7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747 not found: ID does not exist" containerID="7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747"
Oct 31 19:25:26.677108 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.677079    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747"} err="failed to get container status \"7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747\": rpc error: code = NotFound desc = could not find container \"7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747\": container with ID starting with 7fcd90e481b767705a9312aa824fec57f832011c74b98fe9c8e9f4f8d7fe0747 not found: ID does not exist"
Oct 31 19:25:26.828361 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.828334    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"]
Oct 31 19:25:26.832053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.832028    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-dfd4695f9-ggtv6"]
Oct 31 19:25:26.909833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:26.909798    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" path="/var/lib/kubelet/pods/b34fe25e-eb40-40eb-ab9b-c5240dfd6b29/volumes"
Oct 31 19:25:27.548949 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:27.548916    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:27.548949 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:27.548945    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:27.549158 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:27.549039    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:35.549024072 +0000 UTC m=+3838.780262341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:30.458435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:30.458399    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"]
Oct 31 19:25:30.458970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:30.458681    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy" containerID="cri-o://ec8c8c237e17e00c1d1d6f501806bb4c7149e2ecf67e6850a7504197a0761e03" gracePeriod=30
Oct 31 19:25:31.783056 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.783016    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"]
Oct 31 19:25:31.783915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.783894    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" containerName="payload-processing"
Oct 31 19:25:31.783981 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.783919    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" containerName="payload-processing"
Oct 31 19:25:31.784137 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.784125    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34fe25e-eb40-40eb-ab9b-c5240dfd6b29" containerName="payload-processing"
Oct 31 19:25:31.789746 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.789721    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.836643 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.836607    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-credential-socket\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.836826 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.836759    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-data\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.836826 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.836814    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-podinfo\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.837087 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.837057    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-certs\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.837182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.837160    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c299cf17-57da-414c-abb1-99d56c5c96a9-istiod-ca-cert\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.837246 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.837223    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-socket\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.837306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.837258    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh94x\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-kube-api-access-rh94x\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.837371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.837304    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-token\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.837440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.837420    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-envoy\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.846230 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.846203    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"]
Oct 31 19:25:31.889963 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.889930    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-227f0f-openshift-default-dockercfg-l66dj\""
Oct 31 19:25:31.946482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946415    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-certs\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.946716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946522    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c299cf17-57da-414c-abb1-99d56c5c96a9-istiod-ca-cert\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.946716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946576    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-socket\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.946716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946607    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh94x\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-kube-api-access-rh94x\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.946716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946633    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-token\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.946716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946693    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-envoy\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.947032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946757    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-credential-socket\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.947032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946839    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-data\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.947032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.946884    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-podinfo\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.958478 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.948177    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-certs\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.972286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.972252    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-socket\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.980475 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.976443    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-credential-socket\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.980475 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.976996    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-data\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:31.992485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:31.983812    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c299cf17-57da-414c-abb1-99d56c5c96a9-istiod-ca-cert\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:32.029733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.029693    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-envoy\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:32.043582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.040515    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-token\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:32.054087 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.054056    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-podinfo\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:32.054435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.054373    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh94x\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-kube-api-access-rh94x\") pod \"e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") " pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:32.111482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.109779    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:32.562621 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:25:32.562586    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc299cf17_57da_414c_abb1_99d56c5c96a9.slice/crio-dd854ce348964da65d27e4953dc27a8a1f3f76db30dc54792fbfd0cc24b087a6 WatchSource:0}: Error finding container dd854ce348964da65d27e4953dc27a8a1f3f76db30dc54792fbfd0cc24b087a6: Status 404 returned error can't find the container with id dd854ce348964da65d27e4953dc27a8a1f3f76db30dc54792fbfd0cc24b087a6
Oct 31 19:25:32.563316 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.563254    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"]
Oct 31 19:25:32.565779 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.565741    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:32.565876 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.565829    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:32.565913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.565873    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:32.728757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.728721    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" event={"ID":"c299cf17-57da-414c-abb1-99d56c5c96a9","Type":"ContainerStarted","Data":"1bcbfda1df1470784713e52e6212d98f319acada3d9da153f5947429138e3a6d"}
Oct 31 19:25:32.728957 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.728775    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" event={"ID":"c299cf17-57da-414c-abb1-99d56c5c96a9","Type":"ContainerStarted","Data":"dd854ce348964da65d27e4953dc27a8a1f3f76db30dc54792fbfd0cc24b087a6"}
Oct 31 19:25:32.847521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:32.847395    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" podStartSLOduration=1.847374168 podStartE2EDuration="1.847374168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:32.845860154 +0000 UTC m=+3836.077098469" watchObservedRunningTime="2026-10-31 19:25:32.847374168 +0000 UTC m=+3836.078612461"
Oct 31 19:25:33.112141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:33.112065    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:33.118130 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:33.118105    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:33.736902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:33.736870    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:33.737862 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:33.737844    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:25:35.624013 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:35.623811    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:35.624013 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:35.623850    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:35.624013 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:35.623914    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:25:51.623893266 +0000 UTC m=+3854.855131539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:35.781607 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.781069    5551 generic.go:356] "Generic (PLEG): container finished" podID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerID="ec8c8c237e17e00c1d1d6f501806bb4c7149e2ecf67e6850a7504197a0761e03" exitCode=0
Oct 31 19:25:35.786653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.782130    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" event={"ID":"758ecd9d-9d6a-424c-801d-505ffd50f82f","Type":"ContainerDied","Data":"ec8c8c237e17e00c1d1d6f501806bb4c7149e2ecf67e6850a7504197a0761e03"}
Oct 31 19:25:35.786653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.782187    5551 scope.go:122] "RemoveContainer" containerID="e2b476dad947ba6c731449c059c59ac028f86f139c14994d9747385fde2fa13f"
Oct 31 19:25:35.833927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.833907    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:25:35.928009 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.927973    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-socket\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928038    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-certs\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928100    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-podinfo\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928150    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-kube-api-access-tvlww\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928366 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928209    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-envoy\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928366 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928218    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-socket" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:25:35.928366 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928316    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-credential-socket\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928366 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928353    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-certs" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:25:35.928601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928416    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758ecd9d-9d6a-424c-801d-505ffd50f82f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758ecd9d-9d6a-424c-801d-505ffd50f82f-istiod-ca-cert\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928447    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-token\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-token\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928520    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-data\") pod \"758ecd9d-9d6a-424c-801d-505ffd50f82f\" (UID: \"758ecd9d-9d6a-424c-801d-505ffd50f82f\") "
Oct 31 19:25:35.928748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928592    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-credential-socket" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:25:35.928867 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928850    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-data" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:25:35.928927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.928866    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758ecd9d-9d6a-424c-801d-505ffd50f82f-istiod-ca-cert" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:25:35.929085 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.929065    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:35.929164 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.929091    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758ecd9d-9d6a-424c-801d-505ffd50f82f-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:35.929164 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.929106    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:35.929164 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.929119    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:35.929164 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.929132    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:35.931343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.931282    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-token" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:35.931449 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.931383    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-envoy" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:25:35.931449 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.931408    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-kube-api-access-tvlww" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "kube-api-access-tvlww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:35.931576 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:35.931485    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-podinfo" pod "758ecd9d-9d6a-424c-801d-505ffd50f82f" (UID: "758ecd9d-9d6a-424c-801d-505ffd50f82f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:25:36.031854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.031756    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:36.031854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.031798    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:36.031854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.031814    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/758ecd9d-9d6a-424c-801d-505ffd50f82f-kube-api-access-tvlww\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:36.031854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.031829    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758ecd9d-9d6a-424c-801d-505ffd50f82f-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:36.412972 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.412591    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:25:36.412972 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.412691    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:25:36.792956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.792866    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6" event={"ID":"758ecd9d-9d6a-424c-801d-505ffd50f82f","Type":"ContainerDied","Data":"4d50e0af5e2aeff5f3195a9fa8f4c7475f9d246222a33bdbc5c531cfc4b1f5f4"}
Oct 31 19:25:36.792956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.792926    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"
Oct 31 19:25:36.792956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:36.792939    5551 scope.go:122] "RemoveContainer" containerID="ec8c8c237e17e00c1d1d6f501806bb4c7149e2ecf67e6850a7504197a0761e03"
Oct 31 19:25:37.026315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:37.026279    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"]
Oct 31 19:25:37.031515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:37.031476    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3744b0-openshift-default-679cfcfbb9-255f6"]
Oct 31 19:25:38.934258 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:38.934216    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" path="/var/lib/kubelet/pods/758ecd9d-9d6a-424c-801d-505ffd50f82f/volumes"
Oct 31 19:25:40.968104 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:40.968054    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"]
Oct 31 19:25:40.970172 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:40.969521    5551 cpu_manager.go:414] "containerMap: removing container" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy"
Oct 31 19:25:40.970367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:40.970346    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy"
Oct 31 19:25:40.970607 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:40.970592    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy"
Oct 31 19:25:40.970687 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:40.970613    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy"
Oct 31 19:25:40.975518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:40.975497    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029243    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029279    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq22s\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-kube-api-access-jq22s\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029304    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029516    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93479335-6d59-4b1c-b4d4-0a9481688bc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029699 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029676    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029713    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029852 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029828    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029897    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.029975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.029933    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.086503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.081870    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"]
Oct 31 19:25:41.130990 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.130952    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.130990 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131003    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131034    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131064    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131086    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131130    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131146    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq22s\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-kube-api-access-jq22s\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131178    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131235    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93479335-6d59-4b1c-b4d4-0a9481688bc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131524    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131552    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131635    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131801 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131681    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.131965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.131944    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93479335-6d59-4b1c-b4d4-0a9481688bc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.133903 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.133872    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.134143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.134125    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.187515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.187481    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.187694 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.187452    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq22s\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-kube-api-access-jq22s\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hzvrt\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.316344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.316259    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:41.751331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.751295    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"]
Oct 31 19:25:41.755077 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:25:41.755048    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93479335_6d59_4b1c_b4d4_0a9481688bc3.slice/crio-3b5ceddebd6da3f69b15802db5fc0133f714b86662cd3b5d5bc371dc26acb407 WatchSource:0}: Error finding container 3b5ceddebd6da3f69b15802db5fc0133f714b86662cd3b5d5bc371dc26acb407: Status 404 returned error can't find the container with id 3b5ceddebd6da3f69b15802db5fc0133f714b86662cd3b5d5bc371dc26acb407
Oct 31 19:25:41.758872 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.758828    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:41.758974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.758953    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:41.759021 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.759005    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:25:41.846100 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:41.846072    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" event={"ID":"93479335-6d59-4b1c-b4d4-0a9481688bc3","Type":"ContainerStarted","Data":"3b5ceddebd6da3f69b15802db5fc0133f714b86662cd3b5d5bc371dc26acb407"}
Oct 31 19:25:42.888149 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:42.888107    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" event={"ID":"93479335-6d59-4b1c-b4d4-0a9481688bc3","Type":"ContainerStarted","Data":"dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440"}
Oct 31 19:25:43.068483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:43.061631    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podStartSLOduration=3.061611726 podStartE2EDuration="3.061611726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:43.059931339 +0000 UTC m=+3846.291169630" watchObservedRunningTime="2026-10-31 19:25:43.061611726 +0000 UTC m=+3846.292850017"
Oct 31 19:25:43.316525 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:43.316402    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:43.318353 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:43.318323    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hzvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:25:43.318523 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:43.318380    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:25:44.317175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:44.317136    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hzvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:25:44.317617 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:44.317186    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:25:45.317625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:45.317587    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hzvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:25:45.318102 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:45.317646    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:25:46.317078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:46.317030    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hzvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:25:46.317258 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:46.317099    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:25:46.609645 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:46.609544    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:25:47.317499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:47.317416    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hzvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:25:47.317732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:47.317563    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:25:48.316792 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:48.316753    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hzvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:25:48.317319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:48.316827    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:25:49.317505 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:49.317434    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hzvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:25:49.318041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:49.317527    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:25:50.403842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:50.403811    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:50.405087 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:50.405027    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:50.406024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:25:50.406002    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:25:51.722435 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:51.722357    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:51.722435 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:51.722398    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:25:51.723019 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:25:51.722517    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:26:23.722495831 +0000 UTC m=+3886.953734114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:26:04.209282 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.209241    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"]
Oct 31 19:26:04.210128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.210104    5551 cpu_manager.go:414] "containerMap: removing container" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy"
Oct 31 19:26:04.210245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.210129    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758ecd9d-9d6a-424c-801d-505ffd50f82f" containerName="istio-proxy"
Oct 31 19:26:04.214918 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.214896    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.274982 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.274945    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c249cd-openshift-default-dockercfg-q72fc\""
Oct 31 19:26:04.311865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.311826    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-token\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.312191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.312169    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-certs\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.312557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.312533    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htshj\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-kube-api-access-htshj\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.312875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.312855    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-envoy\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.313255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.313076    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istiod-ca-cert\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.313487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.313440    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-podinfo\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.313659 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.313642    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-socket\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.313841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.313821    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-credential-socket\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.313998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.313982    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-data\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.332563 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.332521    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"]
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.436685    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istiod-ca-cert\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.436724    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-podinfo\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.436745    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-socket\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.436785    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-credential-socket\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.436808    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-data\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.436872    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-token\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.436889    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-certs\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.437026    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htshj\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-kube-api-access-htshj\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.437051    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-envoy\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.437755    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-credential-socket\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.438592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.438446    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istiod-ca-cert\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.444849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.440267    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-envoy\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.444849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.440543    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-socket\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.444849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.440766    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-certs\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.444849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.443553    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-data\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.444849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.444169    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-podinfo\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.497739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.497667    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-token\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.497876 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.497796    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htshj\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-kube-api-access-htshj\") pod \"e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") " pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.530669 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.530638    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:04.967247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.967218    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"]
Oct 31 19:26:04.968383 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:26:04.968345    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dd413c_bbce_4a1b_9068_d8bb52df6701.slice/crio-61255991c2f1b25e3daf96f9513426f6401b0d9575fa7d7188ce8bf983bbfdf6 WatchSource:0}: Error finding container 61255991c2f1b25e3daf96f9513426f6401b0d9575fa7d7188ce8bf983bbfdf6: Status 404 returned error can't find the container with id 61255991c2f1b25e3daf96f9513426f6401b0d9575fa7d7188ce8bf983bbfdf6
Oct 31 19:26:04.971089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.971057    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:26:04.971173 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.971128    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:26:04.971173 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:04.971160    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:26:05.063445 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:05.063411    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" event={"ID":"e4dd413c-bbce-4a1b-9068-d8bb52df6701","Type":"ContainerStarted","Data":"44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb"}
Oct 31 19:26:05.063714 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:05.063536    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" event={"ID":"e4dd413c-bbce-4a1b-9068-d8bb52df6701","Type":"ContainerStarted","Data":"61255991c2f1b25e3daf96f9513426f6401b0d9575fa7d7188ce8bf983bbfdf6"}
Oct 31 19:26:06.239399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:06.239337    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podStartSLOduration=2.23932238 podStartE2EDuration="2.23932238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:26:06.233374758 +0000 UTC m=+3869.464613048" watchObservedRunningTime="2026-10-31 19:26:06.23932238 +0000 UTC m=+3869.470560671"
Oct 31 19:26:06.531783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:06.531691    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:06.538276 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:06.538250    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:07.077262 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:07.077233    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:07.078279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:07.078260    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:26:16.607078 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:16.606973    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:26:21.837933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:21.837896    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"]
Oct 31 19:26:21.847243 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:21.847211    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:21.897526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:21.897493    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"]
Oct 31 19:26:21.942032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:21.942000    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c97ce8-serving-cert\""
Oct 31 19:26:21.982745 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:21.982713    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-metrics-certs\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:21.982950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:21.982923    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:21.983046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:21.983023    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/574f9e7c-0781-451a-a0e0-e7800f36f02d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:21.983367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:21.983343    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92jf\" (UniqueName: \"kubernetes.io/projected/574f9e7c-0781-451a-a0e0-e7800f36f02d-kube-api-access-c92jf\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.084747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.084707    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92jf\" (UniqueName: \"kubernetes.io/projected/574f9e7c-0781-451a-a0e0-e7800f36f02d-kube-api-access-c92jf\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.084954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.084795    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-metrics-certs\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.084954 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.084949    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.085128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.085098    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/574f9e7c-0781-451a-a0e0-e7800f36f02d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.085915 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:22.085849    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-serving-cert: secret "maas-api-e2e-inf-a-c97ce8-serving-cert" not found
Oct 31 19:26:22.086063 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:22.085926    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls podName:574f9e7c-0781-451a-a0e0-e7800f36f02d nodeName:}" failed. No retries permitted until 2026-10-31 19:26:22.585908203 +0000 UTC m=+3885.817146485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls") pod "maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5" (UID: "574f9e7c-0781-451a-a0e0-e7800f36f02d") : secret "maas-api-e2e-inf-a-c97ce8-serving-cert" not found
Oct 31 19:26:22.089183 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.087310    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/574f9e7c-0781-451a-a0e0-e7800f36f02d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.096919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.096890    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-metrics-certs\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.142335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.142300    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92jf\" (UniqueName: \"kubernetes.io/projected/574f9e7c-0781-451a-a0e0-e7800f36f02d-kube-api-access-c92jf\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.593433 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.593388    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.596125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.596102    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls\") pod \"maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:22.763395 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:22.763367    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:23.145616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:23.145586    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"]
Oct 31 19:26:23.147594 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:26:23.147566    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574f9e7c_0781_451a_a0e0_e7800f36f02d.slice/crio-a92424d8fc8d4d0659d040f9340de89f0c8dd02cddc688a47c51e8e120275400 WatchSource:0}: Error finding container a92424d8fc8d4d0659d040f9340de89f0c8dd02cddc688a47c51e8e120275400: Status 404 returned error can't find the container with id a92424d8fc8d4d0659d040f9340de89f0c8dd02cddc688a47c51e8e120275400
Oct 31 19:26:23.187531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:23.187490    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5" event={"ID":"574f9e7c-0781-451a-a0e0-e7800f36f02d","Type":"ContainerStarted","Data":"a92424d8fc8d4d0659d040f9340de89f0c8dd02cddc688a47c51e8e120275400"}
Oct 31 19:26:23.809809 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:23.809771    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:26:23.809809 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:23.809804    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:26:23.810065 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:23.809861    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:27:27.809846175 +0000 UTC m=+3951.041084471 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:26:24.199780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:24.199739    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5" event={"ID":"574f9e7c-0781-451a-a0e0-e7800f36f02d","Type":"ContainerStarted","Data":"315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe"}
Oct 31 19:26:24.200220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:24.200034    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:24.328986 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:24.328895    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5" podStartSLOduration=2.3997849860000002 podStartE2EDuration="3.328882148s" totalImagesPullingTime="929.097162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:26:24.325385421 +0000 UTC m=+3887.556623712" watchObservedRunningTime="2026-10-31 19:26:24.328882148 +0000 UTC m=+3887.560120458"
Oct 31 19:26:30.212198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:30.212165    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:26:40.497802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:40.497768    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:26:40.499772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:40.499737    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:26:45.308420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.308388    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"]
Oct 31 19:26:45.313167 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.313143    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.420509 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.420449    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ad2d15-serving-cert\""
Oct 31 19:26:45.425025 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.424994    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"]
Oct 31 19:26:45.436342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.436315    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.436513 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.436368    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m7tn\" (UniqueName: \"kubernetes.io/projected/f6683373-3013-4820-9da0-a16042c77da7-kube-api-access-7m7tn\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.436597 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.436581    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-metrics-certs\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.436805 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.436783    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6683373-3013-4820-9da0-a16042c77da7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.537987 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.537956    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-metrics-certs\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.538161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.538107    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6683373-3013-4820-9da0-a16042c77da7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.538209 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.538170    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.538247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.538208    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m7tn\" (UniqueName: \"kubernetes.io/projected/f6683373-3013-4820-9da0-a16042c77da7-kube-api-access-7m7tn\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.538374 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:45.538343    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-serving-cert: secret "maas-api-e2e-inf-a-ad2d15-serving-cert" not found
Oct 31 19:26:45.538449 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:45.538411    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls podName:f6683373-3013-4820-9da0-a16042c77da7 nodeName:}" failed. No retries permitted until 2026-10-31 19:26:46.038394055 +0000 UTC m=+3909.269632326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls") pod "maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8" (UID: "f6683373-3013-4820-9da0-a16042c77da7") : secret "maas-api-e2e-inf-a-ad2d15-serving-cert" not found
Oct 31 19:26:45.538999 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.538976    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6683373-3013-4820-9da0-a16042c77da7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.540541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.540518    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-metrics-certs\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:45.608316 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:45.608202    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m7tn\" (UniqueName: \"kubernetes.io/projected/f6683373-3013-4820-9da0-a16042c77da7-kube-api-access-7m7tn\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:46.046600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:46.046569    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:46.049254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:46.049231    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:46.235412 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:46.235369    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:46.607302 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:46.607212    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:26:46.650655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:46.650623    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"]
Oct 31 19:26:46.651937 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:26:46.651909    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6683373_3013_4820_9da0_a16042c77da7.slice/crio-c14a406e660883ae1cceb3ac08928db5677e64b02fe1dd5366e3204ba17deeb9 WatchSource:0}: Error finding container c14a406e660883ae1cceb3ac08928db5677e64b02fe1dd5366e3204ba17deeb9: Status 404 returned error can't find the container with id c14a406e660883ae1cceb3ac08928db5677e64b02fe1dd5366e3204ba17deeb9
Oct 31 19:26:47.402023 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:47.401884    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8" event={"ID":"f6683373-3013-4820-9da0-a16042c77da7","Type":"ContainerStarted","Data":"c14a406e660883ae1cceb3ac08928db5677e64b02fe1dd5366e3204ba17deeb9"}
Oct 31 19:26:48.413123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:48.413084    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8" event={"ID":"f6683373-3013-4820-9da0-a16042c77da7","Type":"ContainerStarted","Data":"1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477"}
Oct 31 19:26:48.413684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:48.413296    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:48.530667 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:48.530610    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8" podStartSLOduration=2.698433822 podStartE2EDuration="3.530596005s" totalImagesPullingTime="832.162183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:26:48.527792084 +0000 UTC m=+3911.759030376" watchObservedRunningTime="2026-10-31 19:26:48.530596005 +0000 UTC m=+3911.761834308"
Oct 31 19:26:50.447990 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:50.447899    5551 generic.go:356] "Generic (PLEG): container finished" podID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerID="21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c" exitCode=0
Oct 31 19:26:50.448630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:50.448443    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" event={"ID":"8fa859b3-2ed2-4a73-bfa6-51028ab181b1","Type":"ContainerDied","Data":"21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c"}
Oct 31 19:26:50.448984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:50.448962    5551 scope.go:122] "RemoveContainer" containerID="21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c"
Oct 31 19:26:50.450964 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:50.450932    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:26:50.451087 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:50.451004    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:26:50.451087 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:50.451043    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:26:51.483333 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:51.483287    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" event={"ID":"8fa859b3-2ed2-4a73-bfa6-51028ab181b1","Type":"ContainerStarted","Data":"e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d"}
Oct 31 19:26:52.080935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:52.080896    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:26:52.081141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:52.081075    5551 patch_prober.go:29] interesting pod/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:26:52.081141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:52.081128    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:26:53.079487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:53.078425    5551 patch_prober.go:29] interesting pod/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:26:53.079487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:53.078524    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:26:54.082843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:54.082800    5551 patch_prober.go:29] interesting pod/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:26:54.083254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:54.082870    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:26:54.440363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:54.440337    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:26:55.083241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:55.082816    5551 patch_prober.go:29] interesting pod/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:26:55.083241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:55.082873    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:26:56.080823 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:56.080784    5551 patch_prober.go:29] interesting pod/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:26:56.081109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:56.081079    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:26:57.084104 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:57.084074    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:26:57.084614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:57.084495    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:26:57.085246 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:57.085226    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:26:59.507433 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.507396    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"]
Oct 31 19:26:59.511493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.511425    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.567354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.567323    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"]
Oct 31 19:26:59.600710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.600668    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-metrics-certs\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.600887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.600756    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.600887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.600852    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e578a0-8c6f-47a0-91b6-7c9309e04d86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.601014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.600953    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcgxq\" (UniqueName: \"kubernetes.io/projected/93e578a0-8c6f-47a0-91b6-7c9309e04d86-kube-api-access-xcgxq\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.611827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.611800    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-227f0f-serving-cert\""
Oct 31 19:26:59.702151 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.702115    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.702349 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.702232    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e578a0-8c6f-47a0-91b6-7c9309e04d86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.702349 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:59.702272    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-serving-cert: secret "maas-api-e2e-inf-a-227f0f-serving-cert" not found
Oct 31 19:26:59.702349 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.702333    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcgxq\" (UniqueName: \"kubernetes.io/projected/93e578a0-8c6f-47a0-91b6-7c9309e04d86-kube-api-access-xcgxq\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.702572 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:26:59.702353    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls podName:93e578a0-8c6f-47a0-91b6-7c9309e04d86 nodeName:}" failed. No retries permitted until 2026-10-31 19:27:00.202334946 +0000 UTC m=+3923.433573232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls") pod "maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb" (UID: "93e578a0-8c6f-47a0-91b6-7c9309e04d86") : secret "maas-api-e2e-inf-a-227f0f-serving-cert" not found
Oct 31 19:26:59.702572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.702554    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-metrics-certs\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.702913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.702892    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e578a0-8c6f-47a0-91b6-7c9309e04d86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.705064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.705046    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-metrics-certs\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:26:59.788394 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:26:59.788306    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcgxq\" (UniqueName: \"kubernetes.io/projected/93e578a0-8c6f-47a0-91b6-7c9309e04d86-kube-api-access-xcgxq\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:27:00.212517 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:00.212480    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:27:00.215060 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:00.215034    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls\") pod \"maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:27:00.434563 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:00.434527    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:27:00.818031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:00.818000    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"]
Oct 31 19:27:00.819114 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:27:00.819088    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e578a0_8c6f_47a0_91b6_7c9309e04d86.slice/crio-acb91c2658db23593a37ec3d34dbf706ffc59c9e9f4482378835f526f01982ba WatchSource:0}: Error finding container acb91c2658db23593a37ec3d34dbf706ffc59c9e9f4482378835f526f01982ba: Status 404 returned error can't find the container with id acb91c2658db23593a37ec3d34dbf706ffc59c9e9f4482378835f526f01982ba
Oct 31 19:27:01.565510 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:01.565450    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb" event={"ID":"93e578a0-8c6f-47a0-91b6-7c9309e04d86","Type":"ContainerStarted","Data":"acb91c2658db23593a37ec3d34dbf706ffc59c9e9f4482378835f526f01982ba"}
Oct 31 19:27:02.576211 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:02.576176    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb" event={"ID":"93e578a0-8c6f-47a0-91b6-7c9309e04d86","Type":"ContainerStarted","Data":"f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0"}
Oct 31 19:27:02.576664 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:02.576273    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:27:02.696021 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:02.695964    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb" podStartSLOduration=2.892857914 podStartE2EDuration="3.69594767s" totalImagesPullingTime="803.089756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:02.691995092 +0000 UTC m=+3925.923233386" watchObservedRunningTime="2026-10-31 19:27:02.69594767 +0000 UTC m=+3925.927185959"
Oct 31 19:27:08.593045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:08.592959    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:27:10.689595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:10.689561    5551 patch_prober.go:29] interesting pod/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:27:10.690166 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:10.689625    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:27:12.656925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:12.656892    5551 generic.go:356] "Generic (PLEG): container finished" podID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerID="f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd" exitCode=0
Oct 31 19:27:12.657384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:12.656963    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" event={"ID":"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4","Type":"ContainerDied","Data":"f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd"}
Oct 31 19:27:12.657488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:12.657447    5551 scope.go:122] "RemoveContainer" containerID="f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd"
Oct 31 19:27:12.659212 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:12.659184    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:27:12.659300 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:12.659254    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:27:12.659300 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:12.659284    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:27:13.685020 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:13.684982    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" event={"ID":"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4","Type":"ContainerStarted","Data":"571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323"}
Oct 31 19:27:13.708230 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:13.708201    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:27:13.708496 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:13.708449    5551 patch_prober.go:29] interesting pod/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:27:13.708555 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:13.708532    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:27:14.708624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:14.708590    5551 patch_prober.go:29] interesting pod/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:27:14.709046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:14.708640    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:27:15.708570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:15.708535    5551 patch_prober.go:29] interesting pod/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:27:15.708748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:15.708614    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:27:16.607364 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:27:16.607324    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:27:16.708521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:16.708448    5551 patch_prober.go:29] interesting pod/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:27:16.708711 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:16.708542    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:27:17.709289 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:17.709255    5551 patch_prober.go:29] interesting pod/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:27:17.709725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:17.709308    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:27:18.708864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:18.708829    5551 patch_prober.go:29] interesting pod/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:27:18.709098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:18.708891    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:27:19.712566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:19.712537    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:27:19.713039 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:19.713011    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:27:19.713678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:19.713660    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:27:27.905467 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:27:27.905430    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:27:27.905972 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:27:27.905487    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:27:27.905972 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:27:27.905544    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:29:29.905528181 +0000 UTC m=+4073.136766462 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:27:46.605086 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:27:46.605047    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:27:46.954681 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:46.954654    5551 generic.go:356] "Generic (PLEG): container finished" podID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerID="1bcbfda1df1470784713e52e6212d98f319acada3d9da153f5947429138e3a6d" exitCode=0
Oct 31 19:27:46.954881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:46.954724    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" event={"ID":"c299cf17-57da-414c-abb1-99d56c5c96a9","Type":"ContainerDied","Data":"1bcbfda1df1470784713e52e6212d98f319acada3d9da153f5947429138e3a6d"}
Oct 31 19:27:46.955086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:46.955070    5551 scope.go:122] "RemoveContainer" containerID="1bcbfda1df1470784713e52e6212d98f319acada3d9da153f5947429138e3a6d"
Oct 31 19:27:46.956503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:46.956477    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:27:46.956592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:46.956552    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:27:46.956592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:46.956582    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:27:47.966312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:47.966263    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" event={"ID":"c299cf17-57da-414c-abb1-99d56c5c96a9","Type":"ContainerStarted","Data":"5523361980ebabde5198c9f214144f010ac2621f1f7f3f1b0bd4ca5dc80a59fc"}
Oct 31 19:27:48.111412 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:48.111384    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:27:48.112676 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:48.112653    5551 patch_prober.go:29] interesting pod/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:27:48.112788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:48.112712    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:27:48.737131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:48.737098    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:27:49.111570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:49.111492    5551 patch_prober.go:29] interesting pod/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:27:49.111570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:49.111548    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:27:50.111959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:50.111921    5551 patch_prober.go:29] interesting pod/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:27:50.112521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:50.111985    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:27:51.111904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:51.111865    5551 patch_prober.go:29] interesting pod/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:27:51.112086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:51.111916    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:27:52.112296 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:52.112260    5551 patch_prober.go:29] interesting pod/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:27:52.112719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:52.112316    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:27:53.112182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:53.112135    5551 patch_prober.go:29] interesting pod/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:27:53.112415 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:53.112204    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:27:54.116977 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:54.116939    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:27:54.117825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:54.117807    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:27:58.543827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.543795    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"]
Oct 31 19:27:58.547422 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.547401    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.599318 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.599290    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fdd317-openshift-default-dockercfg-kt7p8\""
Oct 31 19:27:58.652860 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.652824    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"]
Oct 31 19:27:58.666880 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.666845    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-podinfo\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.667076 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.666906    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-token\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.667076 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.667005    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-credential-socket\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.667205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.667168    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-certs\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.667270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.667235    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-data\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.667365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.667348    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-socket\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.667430 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.667380    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjg8\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-kube-api-access-xfjg8\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.667519 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.667500    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a3cc43-225f-4be1-96fb-490ee08eeb04-istiod-ca-cert\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.667577 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.667548    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-envoy\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769212    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-socket\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769254    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjg8\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-kube-api-access-xfjg8\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769495    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a3cc43-225f-4be1-96fb-490ee08eeb04-istiod-ca-cert\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769548    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-envoy\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769647    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-socket\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769679    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-podinfo\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769717    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-token\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769844 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769828    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-credential-socket\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.769937 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769917    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-certs\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.770006 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.769975    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-data\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.770106 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.770079    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a3cc43-225f-4be1-96fb-490ee08eeb04-istiod-ca-cert\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.770171 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.770132    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-credential-socket\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.770403 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.770380    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-certs\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.770585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.770556    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-data\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.772375 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.772348    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-envoy\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.796560 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.796417    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-podinfo\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.827022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.826985    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjg8\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-kube-api-access-xfjg8\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.829399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.829365    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-token\") pod \"e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") " pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:58.860785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:58.860756    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:27:59.301509 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:59.301478    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"]
Oct 31 19:27:59.302980 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:27:59.302940    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a3cc43_225f_4be1_96fb_490ee08eeb04.slice/crio-2b1b1ffc226a874f878b158ab3f2da44adc1043ab9d89760f9959e85a940c920 WatchSource:0}: Error finding container 2b1b1ffc226a874f878b158ab3f2da44adc1043ab9d89760f9959e85a940c920: Status 404 returned error can't find the container with id 2b1b1ffc226a874f878b158ab3f2da44adc1043ab9d89760f9959e85a940c920
Oct 31 19:27:59.305652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:59.305620    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:27:59.305736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:59.305708    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:27:59.305785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:27:59.305736    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:28:00.124091 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:00.124055    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" event={"ID":"86a3cc43-225f-4be1-96fb-490ee08eeb04","Type":"ContainerStarted","Data":"889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f"}
Oct 31 19:28:00.124564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:00.124106    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" event={"ID":"86a3cc43-225f-4be1-96fb-490ee08eeb04","Type":"ContainerStarted","Data":"2b1b1ffc226a874f878b158ab3f2da44adc1043ab9d89760f9959e85a940c920"}
Oct 31 19:28:00.324497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:00.323347    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podStartSLOduration=2.323333545 podStartE2EDuration="2.323333545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:00.323109019 +0000 UTC m=+3983.554347313" watchObservedRunningTime="2026-10-31 19:28:00.323333545 +0000 UTC m=+3983.554571834"
Oct 31 19:28:00.872484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:00.868527    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:28:00.878710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:00.878684    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:28:01.132171 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:01.132035    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:28:01.133439 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:01.133420    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:28:08.119631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:08.119593    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:28:08.120056 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:08.119656    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:28:09.200100 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:09.200064    5551 generic.go:356] "Generic (PLEG): container finished" podID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerID="44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb" exitCode=0
Oct 31 19:28:09.200555 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:09.200136    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" event={"ID":"e4dd413c-bbce-4a1b-9068-d8bb52df6701","Type":"ContainerDied","Data":"44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb"}
Oct 31 19:28:09.200642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:09.200625    5551 scope.go:122] "RemoveContainer" containerID="44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb"
Oct 31 19:28:09.203232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:09.203201    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:28:09.203332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:09.203270    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:28:09.203332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:09.203302    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:28:10.209602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:10.209559    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" event={"ID":"e4dd413c-bbce-4a1b-9068-d8bb52df6701","Type":"ContainerStarted","Data":"cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4"}
Oct 31 19:28:10.531885 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:10.531802    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:28:10.532069 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:10.531993    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 19:28:10.532147 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:10.532074    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 19:28:11.531722 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:11.531680    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 19:28:11.532154 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:11.531749    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 19:28:12.532035 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:12.532000    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 19:28:12.532518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:12.532054    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 19:28:13.531135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:13.531101    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 19:28:13.531333 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:13.531156    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 19:28:14.531722 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:14.531678    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 19:28:14.532265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:14.531731    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 19:28:15.535593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:15.535562    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:28:15.536077 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:15.535974    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:28:15.536562 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:28:15.536542    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:28:16.604511 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:28:16.604475    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:28:46.604810 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:28:46.604776    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:29:16.606435 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:29:16.606351    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:29:29.935721 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:29:29.935690    5551 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c2025e74c640: secret "agc-cred-c2025e74c640" not found
Oct 31 19:29:29.935721 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:29:29.935717    5551 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9: secret "agc-cred-c2025e74c640" not found
Oct 31 19:29:29.936191 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:29:29.935756    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials podName:c2b2f4f7-7e61-44f9-a974-00de58bb47a9 nodeName:}" failed. No retries permitted until 2026-10-31 19:31:31.93574345 +0000 UTC m=+4195.166981719 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials") pod "payload-processing-external-model-686b8c6cf6-56tl9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9") : secret "agc-cred-c2025e74c640" not found
Oct 31 19:29:46.605314 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:29:46.605280    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:29:51.087431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.087392    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"]
Oct 31 19:29:51.087860 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.087604    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8" podUID="f6683373-3013-4820-9da0-a16042c77da7" containerName="maas-api" containerID="cri-o://1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477" gracePeriod=30
Oct 31 19:29:51.385082 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.385050    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:29:51.443767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.443734    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6683373-3013-4820-9da0-a16042c77da7-kube-api-access-7m7tn\" (UniqueName: \"kubernetes.io/projected/f6683373-3013-4820-9da0-a16042c77da7-kube-api-access-7m7tn\") pod \"f6683373-3013-4820-9da0-a16042c77da7\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") "
Oct 31 19:29:51.443948 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.443904    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-metrics-certs\") pod \"f6683373-3013-4820-9da0-a16042c77da7\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") "
Oct 31 19:29:51.443948 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.443932    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6683373-3013-4820-9da0-a16042c77da7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6683373-3013-4820-9da0-a16042c77da7-openshift-service-ca\") pod \"f6683373-3013-4820-9da0-a16042c77da7\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") "
Oct 31 19:29:51.444073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.443974    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls\") pod \"f6683373-3013-4820-9da0-a16042c77da7\" (UID: \"f6683373-3013-4820-9da0-a16042c77da7\") "
Oct 31 19:29:51.444336 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.444303    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6683373-3013-4820-9da0-a16042c77da7-openshift-service-ca" pod "f6683373-3013-4820-9da0-a16042c77da7" (UID: "f6683373-3013-4820-9da0-a16042c77da7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:29:51.444601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.444568    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6683373-3013-4820-9da0-a16042c77da7-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:29:51.446090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.446063    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6683373-3013-4820-9da0-a16042c77da7-kube-api-access-7m7tn" pod "f6683373-3013-4820-9da0-a16042c77da7" (UID: "f6683373-3013-4820-9da0-a16042c77da7"). InnerVolumeSpecName "kube-api-access-7m7tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:29:51.446253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.446233    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls" pod "f6683373-3013-4820-9da0-a16042c77da7" (UID: "f6683373-3013-4820-9da0-a16042c77da7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:29:51.446319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.446266    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-metrics-certs" pod "f6683373-3013-4820-9da0-a16042c77da7" (UID: "f6683373-3013-4820-9da0-a16042c77da7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:29:51.551569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.551489    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:29:51.551569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.551528    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6683373-3013-4820-9da0-a16042c77da7-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:29:51.551569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:51.551542    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m7tn\" (UniqueName: \"kubernetes.io/projected/f6683373-3013-4820-9da0-a16042c77da7-kube-api-access-7m7tn\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:29:52.121966 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.121930    5551 generic.go:356] "Generic (PLEG): container finished" podID="f6683373-3013-4820-9da0-a16042c77da7" containerID="1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477" exitCode=0
Oct 31 19:29:52.136794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.136755    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8" event={"ID":"f6683373-3013-4820-9da0-a16042c77da7","Type":"ContainerDied","Data":"1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477"}
Oct 31 19:29:52.137058 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.137038    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8" event={"ID":"f6683373-3013-4820-9da0-a16042c77da7","Type":"ContainerDied","Data":"c14a406e660883ae1cceb3ac08928db5677e64b02fe1dd5366e3204ba17deeb9"}
Oct 31 19:29:52.137183 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.137170    5551 scope.go:122] "RemoveContainer" containerID="1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477"
Oct 31 19:29:52.137503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.137482    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"
Oct 31 19:29:52.180721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.180694    5551 scope.go:122] "RemoveContainer" containerID="1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477"
Oct 31 19:29:52.181164 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:29:52.181139    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477\": container with ID starting with 1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477 not found: ID does not exist" containerID="1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477"
Oct 31 19:29:52.181275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.181175    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477"} err="failed to get container status \"1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477\": rpc error: code = NotFound desc = could not find container \"1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477\": container with ID starting with 1d6ea4e79d6ba45202a5de1defb85245f120ab39da6eb4774714393d618c9477 not found: ID does not exist"
Oct 31 19:29:52.304180 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.304143    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"]
Oct 31 19:29:52.352437 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.352405    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad2d15-7cb7d94684-4nmj8"]
Oct 31 19:29:52.910028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:52.909996    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6683373-3013-4820-9da0-a16042c77da7" path="/var/lib/kubelet/pods/f6683373-3013-4820-9da0-a16042c77da7/volumes"
Oct 31 19:29:56.694756 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:56.694658    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"]
Oct 31 19:29:56.695441 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:56.695401    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy" containerID="cri-o://571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323" gracePeriod=30
Oct 31 19:29:59.353232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:29:59.353196    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"]
Oct 31 19:30:00.210723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.210685    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"]
Oct 31 19:30:00.211575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.211541    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f6683373-3013-4820-9da0-a16042c77da7" containerName="maas-api"
Oct 31 19:30:00.211575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.211564    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6683373-3013-4820-9da0-a16042c77da7" containerName="maas-api"
Oct 31 19:30:00.211800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.211753    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6683373-3013-4820-9da0-a16042c77da7" containerName="maas-api"
Oct 31 19:30:00.226481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.223804    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:00.285501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.270683    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"]
Oct 31 19:30:00.285501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.276035    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:00.335582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.331565    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"]
Oct 31 19:30:00.365772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.356765    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:00.365772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.358400    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:30:00.365772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.358691    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkvh2\" (UniqueName: \"kubernetes.io/projected/478f858f-3c8c-44df-b97a-3aa8c14a0e86-kube-api-access-nkvh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv\" (UID: \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:00.365772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.358766    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd7ec9a-60e8-4553-afb6-aac6e383a844-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7\" (UID: \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:00.365772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.359007    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzkv\" (UniqueName: \"kubernetes.io/projected/6dd7ec9a-60e8-4553-afb6-aac6e383a844-kube-api-access-cxzkv\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7\" (UID: \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:00.365772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.359055    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/478f858f-3c8c-44df-b97a-3aa8c14a0e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv\" (UID: \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:00.389480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.369847    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"]
Oct 31 19:30:00.391557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.390725    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:00.413089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.413020    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"]
Oct 31 19:30:00.419842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.419821    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"]
Oct 31 19:30:00.432589 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.432565    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"]
Oct 31 19:30:00.448792 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.448764    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"]
Oct 31 19:30:00.460362 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.460326    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb\" (UID: \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:00.460674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.460649    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txp5k\" (UniqueName: \"kubernetes.io/projected/59e70a9c-2192-4149-af18-38d5885d6f3f-kube-api-access-txp5k\") pod \"maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk\" (UID: \"59e70a9c-2192-4149-af18-38d5885d6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:00.460961 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.460911    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzkv\" (UniqueName: \"kubernetes.io/projected/6dd7ec9a-60e8-4553-afb6-aac6e383a844-kube-api-access-cxzkv\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7\" (UID: \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:00.461428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.461397    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/478f858f-3c8c-44df-b97a-3aa8c14a0e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv\" (UID: \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:00.462472 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.462434    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkvh2\" (UniqueName: \"kubernetes.io/projected/478f858f-3c8c-44df-b97a-3aa8c14a0e86-kube-api-access-nkvh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv\" (UID: \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:00.462898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.462876    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e70a9c-2192-4149-af18-38d5885d6f3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk\" (UID: \"59e70a9c-2192-4149-af18-38d5885d6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:00.463096 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.463079    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd7ec9a-60e8-4553-afb6-aac6e383a844-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7\" (UID: \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:00.463991 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.463972    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxjxv\" (UniqueName: \"kubernetes.io/projected/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-kube-api-access-gxjxv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb\" (UID: \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:00.464280 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.463890    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd7ec9a-60e8-4553-afb6-aac6e383a844-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7\" (UID: \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:00.464412 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.462286    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/478f858f-3c8c-44df-b97a-3aa8c14a0e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv\" (UID: \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:00.518761 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.518727    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzkv\" (UniqueName: \"kubernetes.io/projected/6dd7ec9a-60e8-4553-afb6-aac6e383a844-kube-api-access-cxzkv\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7\" (UID: \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:00.519009 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.518834    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkvh2\" (UniqueName: \"kubernetes.io/projected/478f858f-3c8c-44df-b97a-3aa8c14a0e86-kube-api-access-nkvh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv\" (UID: \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:00.568123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.566159    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e70a9c-2192-4149-af18-38d5885d6f3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk\" (UID: \"59e70a9c-2192-4149-af18-38d5885d6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:00.568123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.566254    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjxv\" (UniqueName: \"kubernetes.io/projected/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-kube-api-access-gxjxv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb\" (UID: \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:00.568123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.566405    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb\" (UID: \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:00.568123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.566439    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txp5k\" (UniqueName: \"kubernetes.io/projected/59e70a9c-2192-4149-af18-38d5885d6f3f-kube-api-access-txp5k\") pod \"maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk\" (UID: \"59e70a9c-2192-4149-af18-38d5885d6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:00.569495 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.568807    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb\" (UID: \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:00.569495 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.569435    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e70a9c-2192-4149-af18-38d5885d6f3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk\" (UID: \"59e70a9c-2192-4149-af18-38d5885d6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:00.624272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.624234    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txp5k\" (UniqueName: \"kubernetes.io/projected/59e70a9c-2192-4149-af18-38d5885d6f3f-kube-api-access-txp5k\") pod \"maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk\" (UID: \"59e70a9c-2192-4149-af18-38d5885d6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:00.625354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.625318    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjxv\" (UniqueName: \"kubernetes.io/projected/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-kube-api-access-gxjxv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb\" (UID: \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:00.669162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.668037    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:00.730285 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.730189    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:00.734268 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.734236    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:00.737545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:00.737518    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:01.106105 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:01.106077    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"]
Oct 31 19:30:01.106528 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:30:01.106497    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478f858f_3c8c_44df_b97a_3aa8c14a0e86.slice/crio-89f3fea6db2333f4366e804c46ed59c589cd78c62ab0fd7500f94e26915ae729 WatchSource:0}: Error finding container 89f3fea6db2333f4366e804c46ed59c589cd78c62ab0fd7500f94e26915ae729: Status 404 returned error can't find the container with id 89f3fea6db2333f4366e804c46ed59c589cd78c62ab0fd7500f94e26915ae729
Oct 31 19:30:01.232957 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:01.232920    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"]
Oct 31 19:30:01.290486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:01.290278    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" event={"ID":"59e70a9c-2192-4149-af18-38d5885d6f3f","Type":"ContainerStarted","Data":"1efed8a6c3c24724a49b698e95035f2b1c37b6b4f7188e74a496b2f459b17089"}
Oct 31 19:30:01.291390 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:01.291328    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"]
Oct 31 19:30:01.292641 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:30:01.292616    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c3782c_02a6_4fda_8fdc_0d423cb9f39b.slice/crio-abb62f4b318408ce97089c946ac4f3db2ab9e6ccf0e971bf1fb37b255e072f34 WatchSource:0}: Error finding container abb62f4b318408ce97089c946ac4f3db2ab9e6ccf0e971bf1fb37b255e072f34: Status 404 returned error can't find the container with id abb62f4b318408ce97089c946ac4f3db2ab9e6ccf0e971bf1fb37b255e072f34
Oct 31 19:30:01.293248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:01.293219    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" event={"ID":"478f858f-3c8c-44df-b97a-3aa8c14a0e86","Type":"ContainerStarted","Data":"89f3fea6db2333f4366e804c46ed59c589cd78c62ab0fd7500f94e26915ae729"}
Oct 31 19:30:01.296713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:01.296606    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"]
Oct 31 19:30:02.094487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.091950    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:30:02.177655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.177617    5551 patch_prober.go:29] interesting pod/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:30:02.177952 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.177922    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.209541    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-kube-api-access-9kmmn\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-kube-api-access-9kmmn\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.209607    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-certs\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.209924    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-token\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.209996    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-envoy\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.210032    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-data\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.210107    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-credential-socket\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.210143    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istiod-ca-cert\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.210255    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-podinfo\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.210500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.210354    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-socket\") pod \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\" (UID: \"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4\") "
Oct 31 19:30:02.213099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.212679    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-data" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:02.213099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.212798    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istiod-ca-cert" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:02.213099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.213057    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-credential-socket" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:02.213367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.213108    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-socket" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:02.213557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.213520    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-certs" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:02.222396 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.222363    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-kube-api-access-9kmmn" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "kube-api-access-9kmmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:02.222829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.222797    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-token" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:02.223907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.223884    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.224019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.223911    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.224019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.223925    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.224019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.223937    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.224019 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.223951    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.224288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.224259    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-podinfo" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:30:02.224369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.224349    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-envoy" pod "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" (UID: "5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:02.306737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.306653    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"]
Oct 31 19:30:02.306994 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.306958    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb" podUID="93e578a0-8c6f-47a0-91b6-7c9309e04d86" containerName="maas-api" containerID="cri-o://f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0" gracePeriod=30
Oct 31 19:30:02.312689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.312658    5551 generic.go:356] "Generic (PLEG): container finished" podID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerID="571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323" exitCode=0
Oct 31 19:30:02.313033 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.313004    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" event={"ID":"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4","Type":"ContainerDied","Data":"571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323"}
Oct 31 19:30:02.313129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.313071    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6" event={"ID":"5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4","Type":"ContainerDied","Data":"bb1998fdf41aae6e5138404d481865fcf2f9ae551f731dfda5991d4a262a7d67"}
Oct 31 19:30:02.313129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.313105    5551 scope.go:122] "RemoveContainer" containerID="571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323"
Oct 31 19:30:02.313295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.313277    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"
Oct 31 19:30:02.326976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.326939    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kmmn\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-kube-api-access-9kmmn\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.326976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.326979    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.327237 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.327016    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.327237 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.327030    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:02.337929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.337882    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" event={"ID":"6dd7ec9a-60e8-4553-afb6-aac6e383a844","Type":"ContainerStarted","Data":"82a0a5d107118c36464e3683600b4eebd00c3c3e6f6ff66f8d21cc60467c1f3d"}
Oct 31 19:30:02.338088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.337962    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" event={"ID":"6dd7ec9a-60e8-4553-afb6-aac6e383a844","Type":"ContainerStarted","Data":"fb97c8fb7b2a8bb919ee6ae1350c40ce3ec1c56f50bfd9affbe525c25b54ab8a"}
Oct 31 19:30:02.341701 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.341647    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" event={"ID":"59e70a9c-2192-4149-af18-38d5885d6f3f","Type":"ContainerStarted","Data":"af41ad3b4b8876243f169c7f5d62976a600dfe464a31ecd85a49852f0d0817bd"}
Oct 31 19:30:02.344946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.344896    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" event={"ID":"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b","Type":"ContainerStarted","Data":"21e781737313e2d47971bcc4540ec5e6d402d352085c014cf1b3396525a9f2ee"}
Oct 31 19:30:02.345063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.344971    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" event={"ID":"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b","Type":"ContainerStarted","Data":"abb62f4b318408ce97089c946ac4f3db2ab9e6ccf0e971bf1fb37b255e072f34"}
Oct 31 19:30:02.347048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.347023    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" event={"ID":"478f858f-3c8c-44df-b97a-3aa8c14a0e86","Type":"ContainerStarted","Data":"154cff5336b8c920ddf2d2dfaef5d43fd2322821dc4c1ce77546cbea8595964d"}
Oct 31 19:30:02.347435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.347413    5551 scope.go:122] "RemoveContainer" containerID="154cff5336b8c920ddf2d2dfaef5d43fd2322821dc4c1ce77546cbea8595964d"
Oct 31 19:30:02.387700 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.387600    5551 scope.go:122] "RemoveContainer" containerID="f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd"
Oct 31 19:30:02.443135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.443011    5551 scope.go:122] "RemoveContainer" containerID="571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323"
Oct 31 19:30:02.443711 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:02.443572    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323\": container with ID starting with 571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323 not found: ID does not exist" containerID="571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323"
Oct 31 19:30:02.443711 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.443615    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323"} err="failed to get container status \"571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323\": rpc error: code = NotFound desc = could not find container \"571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323\": container with ID starting with 571c252dbdd0b2548d9938e11ba6b8678dea9f3af7a81d469ecdbf1b29241323 not found: ID does not exist"
Oct 31 19:30:02.443711 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.443640    5551 scope.go:122] "RemoveContainer" containerID="f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd"
Oct 31 19:30:02.444342 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:02.444282    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd\": container with ID starting with f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd not found: ID does not exist" containerID="f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd"
Oct 31 19:30:02.444342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.444312    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd"} err="failed to get container status \"f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd\": rpc error: code = NotFound desc = could not find container \"f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd\": container with ID starting with f931c707e4a8d6040aecea73ef177d934b0ad54d6d708194b2bb45b3fa09d4dd not found: ID does not exist"
Oct 31 19:30:02.541937 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.541882    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"]
Oct 31 19:30:02.548906 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.548814    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ad2d15-openshift-default-6b94764dfc-795w6"]
Oct 31 19:30:02.866710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.866659    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" podStartSLOduration=2.866640267 podStartE2EDuration="2.866640267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:02.863039752 +0000 UTC m=+4106.094278039" watchObservedRunningTime="2026-10-31 19:30:02.866640267 +0000 UTC m=+4106.097878558"
Oct 31 19:30:02.888345 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.888312    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:30:02.912934 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.912903    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" path="/var/lib/kubelet/pods/5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4/volumes"
Oct 31 19:30:02.984313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.983488    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e578a0-8c6f-47a0-91b6-7c9309e04d86-kube-api-access-xcgxq\" (UniqueName: \"kubernetes.io/projected/93e578a0-8c6f-47a0-91b6-7c9309e04d86-kube-api-access-xcgxq\") pod \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") "
Oct 31 19:30:02.984313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.983564    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls\") pod \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") "
Oct 31 19:30:02.984313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.983710    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e578a0-8c6f-47a0-91b6-7c9309e04d86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e578a0-8c6f-47a0-91b6-7c9309e04d86-openshift-service-ca\") pod \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") "
Oct 31 19:30:02.984313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.983765    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-metrics-certs\") pod \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\" (UID: \"93e578a0-8c6f-47a0-91b6-7c9309e04d86\") "
Oct 31 19:30:02.990107 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.989652    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-metrics-certs" pod "93e578a0-8c6f-47a0-91b6-7c9309e04d86" (UID: "93e578a0-8c6f-47a0-91b6-7c9309e04d86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:30:02.994613 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:02.993005    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e578a0-8c6f-47a0-91b6-7c9309e04d86-openshift-service-ca" pod "93e578a0-8c6f-47a0-91b6-7c9309e04d86" (UID: "93e578a0-8c6f-47a0-91b6-7c9309e04d86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:03.007012 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.006953    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" podStartSLOduration=3.006917077 podStartE2EDuration="3.006917077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:02.978370394 +0000 UTC m=+4106.209608696" watchObservedRunningTime="2026-10-31 19:30:03.006917077 +0000 UTC m=+4106.238155368"
Oct 31 19:30:03.015103 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.013786    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e578a0-8c6f-47a0-91b6-7c9309e04d86-kube-api-access-xcgxq" pod "93e578a0-8c6f-47a0-91b6-7c9309e04d86" (UID: "93e578a0-8c6f-47a0-91b6-7c9309e04d86"). InnerVolumeSpecName "kube-api-access-xcgxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:03.015103 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.013861    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls" pod "93e578a0-8c6f-47a0-91b6-7c9309e04d86" (UID: "93e578a0-8c6f-47a0-91b6-7c9309e04d86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:30:03.087135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.086905    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcgxq\" (UniqueName: \"kubernetes.io/projected/93e578a0-8c6f-47a0-91b6-7c9309e04d86-kube-api-access-xcgxq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:03.087135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.086950    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:03.087135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.086967    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e578a0-8c6f-47a0-91b6-7c9309e04d86-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:03.087135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.086982    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e578a0-8c6f-47a0-91b6-7c9309e04d86-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:03.094257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.094202    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" podStartSLOduration=3.094185902 podStartE2EDuration="3.094185902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.090656075 +0000 UTC m=+4106.321894366" watchObservedRunningTime="2026-10-31 19:30:03.094185902 +0000 UTC m=+4106.325424191"
Oct 31 19:30:03.437524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.433176    5551 generic.go:356] "Generic (PLEG): container finished" podID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerID="82a0a5d107118c36464e3683600b4eebd00c3c3e6f6ff66f8d21cc60467c1f3d" exitCode=60
Oct 31 19:30:03.437524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.434353    5551 scope.go:122] "RemoveContainer" containerID="82a0a5d107118c36464e3683600b4eebd00c3c3e6f6ff66f8d21cc60467c1f3d"
Oct 31 19:30:03.437524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.434852    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" event={"ID":"6dd7ec9a-60e8-4553-afb6-aac6e383a844","Type":"ContainerDied","Data":"82a0a5d107118c36464e3683600b4eebd00c3c3e6f6ff66f8d21cc60467c1f3d"}
Oct 31 19:30:03.456723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.449630    5551 generic.go:356] "Generic (PLEG): container finished" podID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerID="af41ad3b4b8876243f169c7f5d62976a600dfe464a31ecd85a49852f0d0817bd" exitCode=60
Oct 31 19:30:03.456723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.449717    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" event={"ID":"59e70a9c-2192-4149-af18-38d5885d6f3f","Type":"ContainerDied","Data":"af41ad3b4b8876243f169c7f5d62976a600dfe464a31ecd85a49852f0d0817bd"}
Oct 31 19:30:03.456723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.450480    5551 scope.go:122] "RemoveContainer" containerID="af41ad3b4b8876243f169c7f5d62976a600dfe464a31ecd85a49852f0d0817bd"
Oct 31 19:30:03.456723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.456141    5551 generic.go:356] "Generic (PLEG): container finished" podID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerID="21e781737313e2d47971bcc4540ec5e6d402d352085c014cf1b3396525a9f2ee" exitCode=60
Oct 31 19:30:03.456723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.456250    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" event={"ID":"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b","Type":"ContainerDied","Data":"21e781737313e2d47971bcc4540ec5e6d402d352085c014cf1b3396525a9f2ee"}
Oct 31 19:30:03.456723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.456693    5551 scope.go:122] "RemoveContainer" containerID="21e781737313e2d47971bcc4540ec5e6d402d352085c014cf1b3396525a9f2ee"
Oct 31 19:30:03.468703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.462246    5551 generic.go:356] "Generic (PLEG): container finished" podID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerID="154cff5336b8c920ddf2d2dfaef5d43fd2322821dc4c1ce77546cbea8595964d" exitCode=60
Oct 31 19:30:03.468703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.462366    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" event={"ID":"478f858f-3c8c-44df-b97a-3aa8c14a0e86","Type":"ContainerDied","Data":"154cff5336b8c920ddf2d2dfaef5d43fd2322821dc4c1ce77546cbea8595964d"}
Oct 31 19:30:03.468703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.462428    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" event={"ID":"478f858f-3c8c-44df-b97a-3aa8c14a0e86","Type":"ContainerStarted","Data":"b4c0a8ea4f47d2002422bf41b12819476e93c26ac2a97b6f86f7ed04e0bb5be6"}
Oct 31 19:30:03.509616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.509552    5551 generic.go:356] "Generic (PLEG): container finished" podID="93e578a0-8c6f-47a0-91b6-7c9309e04d86" containerID="f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0" exitCode=0
Oct 31 19:30:03.509781 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.509618    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb" event={"ID":"93e578a0-8c6f-47a0-91b6-7c9309e04d86","Type":"ContainerDied","Data":"f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0"}
Oct 31 19:30:03.509781 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.509642    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"
Oct 31 19:30:03.509781 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.509686    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb" event={"ID":"93e578a0-8c6f-47a0-91b6-7c9309e04d86","Type":"ContainerDied","Data":"acb91c2658db23593a37ec3d34dbf706ffc59c9e9f4482378835f526f01982ba"}
Oct 31 19:30:03.509781 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.509717    5551 scope.go:122] "RemoveContainer" containerID="f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0"
Oct 31 19:30:03.536217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.536192    5551 scope.go:122] "RemoveContainer" containerID="f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0"
Oct 31 19:30:03.536688 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:03.536662    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0\": container with ID starting with f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0 not found: ID does not exist" containerID="f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0"
Oct 31 19:30:03.536789 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.536700    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0"} err="failed to get container status \"f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0\": rpc error: code = NotFound desc = could not find container \"f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0\": container with ID starting with f683b820144c4b65b91a70f0a2412f6b9c6546091897c03708650ade517033a0 not found: ID does not exist"
Oct 31 19:30:03.688172 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:03.688058    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" podStartSLOduration=3.68804466 podStartE2EDuration="3.68804466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.685977632 +0000 UTC m=+4106.917215923" watchObservedRunningTime="2026-10-31 19:30:03.68804466 +0000 UTC m=+4106.919282949"
Oct 31 19:30:04.026737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.026702    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"]
Oct 31 19:30:04.029978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.029949    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-227f0f-6cfc5b99b6-rwrtb"]
Oct 31 19:30:04.354795 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.354698    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9" podUID="c2b2f4f7-7e61-44f9-a974-00de58bb47a9" containerName="payload-processing" containerID="cri-o://a22d982ff6525fc2ac3928dc8bc97e8b4650a4379e3cf120bfecab30c3fc4a0d" gracePeriod=25
Oct 31 19:30:04.522679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.521386    5551 generic.go:356] "Generic (PLEG): container finished" podID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerID="fdb3e6d46224e2ecaf1c0d9f78d1bd214fb95c871d409ec4196615778699769b" exitCode=60
Oct 31 19:30:04.522679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.521505    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" event={"ID":"6dd7ec9a-60e8-4553-afb6-aac6e383a844","Type":"ContainerDied","Data":"fdb3e6d46224e2ecaf1c0d9f78d1bd214fb95c871d409ec4196615778699769b"}
Oct 31 19:30:04.522679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.521573    5551 scope.go:122] "RemoveContainer" containerID="82a0a5d107118c36464e3683600b4eebd00c3c3e6f6ff66f8d21cc60467c1f3d"
Oct 31 19:30:04.522679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.521868    5551 scope.go:122] "RemoveContainer" containerID="fdb3e6d46224e2ecaf1c0d9f78d1bd214fb95c871d409ec4196615778699769b"
Oct 31 19:30:04.522679 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:04.522205    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7_odh-ai-gateway-infra(6dd7ec9a-60e8-4553-afb6-aac6e383a844)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844"
Oct 31 19:30:04.524500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.524238    5551 generic.go:356] "Generic (PLEG): container finished" podID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerID="889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f" exitCode=0
Oct 31 19:30:04.524500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.524320    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" event={"ID":"86a3cc43-225f-4be1-96fb-490ee08eeb04","Type":"ContainerDied","Data":"889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f"}
Oct 31 19:30:04.524747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.524728    5551 scope.go:122] "RemoveContainer" containerID="889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f"
Oct 31 19:30:04.526822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.526789    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:30:04.526930 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.526880    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:30:04.526969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.526952    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:30:04.526969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.526953    5551 generic.go:356] "Generic (PLEG): container finished" podID="c2b2f4f7-7e61-44f9-a974-00de58bb47a9" containerID="a22d982ff6525fc2ac3928dc8bc97e8b4650a4379e3cf120bfecab30c3fc4a0d" exitCode=0
Oct 31 19:30:04.527061 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.527043    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9" event={"ID":"c2b2f4f7-7e61-44f9-a974-00de58bb47a9","Type":"ContainerDied","Data":"a22d982ff6525fc2ac3928dc8bc97e8b4650a4379e3cf120bfecab30c3fc4a0d"}
Oct 31 19:30:04.529832 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.529804    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" event={"ID":"59e70a9c-2192-4149-af18-38d5885d6f3f","Type":"ContainerStarted","Data":"3c8ddb164529560f084f3db6b7ecfb3c604f73440cc587f045feef5ed6b2cae5"}
Oct 31 19:30:04.532523 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.532495    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" event={"ID":"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b","Type":"ContainerStarted","Data":"0f20d8144775b146994f11680d85f57e716424ad9dcbf69ebf296bc5b37958c7"}
Oct 31 19:30:04.547076 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.536898    5551 generic.go:356] "Generic (PLEG): container finished" podID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerID="b4c0a8ea4f47d2002422bf41b12819476e93c26ac2a97b6f86f7ed04e0bb5be6" exitCode=60
Oct 31 19:30:04.547076 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.537048    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" event={"ID":"478f858f-3c8c-44df-b97a-3aa8c14a0e86","Type":"ContainerDied","Data":"b4c0a8ea4f47d2002422bf41b12819476e93c26ac2a97b6f86f7ed04e0bb5be6"}
Oct 31 19:30:04.547076 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.537084    5551 scope.go:122] "RemoveContainer" containerID="154cff5336b8c920ddf2d2dfaef5d43fd2322821dc4c1ce77546cbea8595964d"
Oct 31 19:30:04.547076 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.537359    5551 scope.go:122] "RemoveContainer" containerID="b4c0a8ea4f47d2002422bf41b12819476e93c26ac2a97b6f86f7ed04e0bb5be6"
Oct 31 19:30:04.547076 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:04.537633    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv_odh-ai-gateway-infra(478f858f-3c8c-44df-b97a-3aa8c14a0e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86"
Oct 31 19:30:04.648960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.648932    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:30:04.740767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.740738    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials\") pod \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") "
Oct 31 19:30:04.740919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.740816    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-kube-api-access-p9rgf\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-kube-api-access-p9rgf\") pod \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") "
Oct 31 19:30:04.740919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.740872    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-plugins-config-volume\") pod \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") "
Oct 31 19:30:04.741046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.740920    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-routing-overlay-volume\") pod \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\" (UID: \"c2b2f4f7-7e61-44f9-a974-00de58bb47a9\") "
Oct 31 19:30:04.741400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.741370    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-plugins-config-volume" pod "c2b2f4f7-7e61-44f9-a974-00de58bb47a9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:04.741536 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.741441    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-routing-overlay-volume" pod "c2b2f4f7-7e61-44f9-a974-00de58bb47a9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:04.743123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.743095    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-kube-api-access-p9rgf" pod "c2b2f4f7-7e61-44f9-a974-00de58bb47a9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9"). InnerVolumeSpecName "kube-api-access-p9rgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:04.743618 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.743490    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials" pod "c2b2f4f7-7e61-44f9-a974-00de58bb47a9" (UID: "c2b2f4f7-7e61-44f9-a974-00de58bb47a9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:04.842806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.842772    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:04.842806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.842799    5551 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-routing-overlay-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:04.842806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.842810    5551 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-provider-credentials\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:04.843039 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.842819    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9rgf\" (UniqueName: \"kubernetes.io/projected/c2b2f4f7-7e61-44f9-a974-00de58bb47a9-kube-api-access-p9rgf\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:04.910266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:04.910234    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e578a0-8c6f-47a0-91b6-7c9309e04d86" path="/var/lib/kubelet/pods/93e578a0-8c6f-47a0-91b6-7c9309e04d86/volumes"
Oct 31 19:30:05.556889 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.556853    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9" event={"ID":"c2b2f4f7-7e61-44f9-a974-00de58bb47a9","Type":"ContainerDied","Data":"28a65376b91de63f8cd11e263fc90ee5fb9c88a4f74fa54f389fdf7c4909420e"}
Oct 31 19:30:05.557568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.557548    5551 scope.go:122] "RemoveContainer" containerID="a22d982ff6525fc2ac3928dc8bc97e8b4650a4379e3cf120bfecab30c3fc4a0d"
Oct 31 19:30:05.557750 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.557477    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"
Oct 31 19:30:05.564058 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.564033    5551 generic.go:356] "Generic (PLEG): container finished" podID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerID="3c8ddb164529560f084f3db6b7ecfb3c604f73440cc587f045feef5ed6b2cae5" exitCode=60
Oct 31 19:30:05.564707 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.564599    5551 scope.go:122] "RemoveContainer" containerID="3c8ddb164529560f084f3db6b7ecfb3c604f73440cc587f045feef5ed6b2cae5"
Oct 31 19:30:05.565210 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:05.565185    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk_odh-ai-gateway-infra(59e70a9c-2192-4149-af18-38d5885d6f3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f"
Oct 31 19:30:05.565579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.564617    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" event={"ID":"59e70a9c-2192-4149-af18-38d5885d6f3f","Type":"ContainerDied","Data":"3c8ddb164529560f084f3db6b7ecfb3c604f73440cc587f045feef5ed6b2cae5"}
Oct 31 19:30:05.568187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.568166    5551 generic.go:356] "Generic (PLEG): container finished" podID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerID="0f20d8144775b146994f11680d85f57e716424ad9dcbf69ebf296bc5b37958c7" exitCode=60
Oct 31 19:30:05.568411 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.568393    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" event={"ID":"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b","Type":"ContainerDied","Data":"0f20d8144775b146994f11680d85f57e716424ad9dcbf69ebf296bc5b37958c7"}
Oct 31 19:30:05.568939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.568921    5551 scope.go:122] "RemoveContainer" containerID="0f20d8144775b146994f11680d85f57e716424ad9dcbf69ebf296bc5b37958c7"
Oct 31 19:30:05.569869 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:05.569378    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb_odh-ai-gateway-infra(c7c3782c-02a6-4fda-8fdc-0d423cb9f39b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b"
Oct 31 19:30:05.572505 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.572484    5551 scope.go:122] "RemoveContainer" containerID="b4c0a8ea4f47d2002422bf41b12819476e93c26ac2a97b6f86f7ed04e0bb5be6"
Oct 31 19:30:05.575759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.575347    5551 scope.go:122] "RemoveContainer" containerID="fdb3e6d46224e2ecaf1c0d9f78d1bd214fb95c871d409ec4196615778699769b"
Oct 31 19:30:05.575759 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:05.575694    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7_odh-ai-gateway-infra(6dd7ec9a-60e8-4553-afb6-aac6e383a844)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844"
Oct 31 19:30:05.577825 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:05.577804    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv_odh-ai-gateway-infra(478f858f-3c8c-44df-b97a-3aa8c14a0e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86"
Oct 31 19:30:05.578729 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.578681    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" event={"ID":"86a3cc43-225f-4be1-96fb-490ee08eeb04","Type":"ContainerStarted","Data":"bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1"}
Oct 31 19:30:05.597094 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.597068    5551 scope.go:122] "RemoveContainer" containerID="af41ad3b4b8876243f169c7f5d62976a600dfe464a31ecd85a49852f0d0817bd"
Oct 31 19:30:05.612644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.612605    5551 scope.go:122] "RemoveContainer" containerID="21e781737313e2d47971bcc4540ec5e6d402d352085c014cf1b3396525a9f2ee"
Oct 31 19:30:05.725569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.725527    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"]
Oct 31 19:30:05.728803 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.728781    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b8c6cf6-56tl9"]
Oct 31 19:30:05.868287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.868200    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:30:05.868992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.868654    5551 patch_prober.go:29] interesting pod/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:30:05.868992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:05.868714    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:30:06.592672 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:06.592647    5551 scope.go:122] "RemoveContainer" containerID="3c8ddb164529560f084f3db6b7ecfb3c604f73440cc587f045feef5ed6b2cae5"
Oct 31 19:30:06.593598 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:06.593570    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk_odh-ai-gateway-infra(59e70a9c-2192-4149-af18-38d5885d6f3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f"
Oct 31 19:30:06.594345 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:06.594323    5551 scope.go:122] "RemoveContainer" containerID="0f20d8144775b146994f11680d85f57e716424ad9dcbf69ebf296bc5b37958c7"
Oct 31 19:30:06.594728 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:06.594704    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb_odh-ai-gateway-infra(c7c3782c-02a6-4fda-8fdc-0d423cb9f39b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b"
Oct 31 19:30:06.864539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:06.862815    5551 patch_prober.go:29] interesting pod/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:30:06.864539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:06.862873    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:30:06.910745 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:06.910713    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b2f4f7-7e61-44f9-a974-00de58bb47a9" path="/var/lib/kubelet/pods/c2b2f4f7-7e61-44f9-a974-00de58bb47a9/volumes"
Oct 31 19:30:07.863483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:07.862838    5551 patch_prober.go:29] interesting pod/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:30:07.863483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:07.862905    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:30:08.874559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:08.873783    5551 patch_prober.go:29] interesting pod/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:30:08.874559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:08.873844    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:30:09.433489 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:09.433115    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"]
Oct 31 19:30:09.443585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:09.433396    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy" containerID="cri-o://5523361980ebabde5198c9f214144f010ac2621f1f7f3f1b0bd4ca5dc80a59fc" gracePeriod=30
Oct 31 19:30:09.870645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:09.870556    5551 patch_prober.go:29] interesting pod/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:30:09.870925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:09.870892    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:30:10.861115 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:10.861072    5551 patch_prober.go:29] interesting pod/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:30:10.861526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:10.861148    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:30:11.865939 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:11.865899    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:30:11.866325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:11.866182    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:30:11.866842 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:11.866824    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:30:13.902775 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:13.902743    5551 scope.go:122] "RemoveContainer" containerID="b4c0a8ea4f47d2002422bf41b12819476e93c26ac2a97b6f86f7ed04e0bb5be6"
Oct 31 19:30:14.656194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.656155    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" event={"ID":"478f858f-3c8c-44df-b97a-3aa8c14a0e86","Type":"ContainerStarted","Data":"0a418611fa11c83452969c624b403641240ac687610ccaba91dd800c7634b57f"}
Oct 31 19:30:14.658225 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.658198    5551 generic.go:356] "Generic (PLEG): container finished" podID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerID="5523361980ebabde5198c9f214144f010ac2621f1f7f3f1b0bd4ca5dc80a59fc" exitCode=0
Oct 31 19:30:14.658389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.658269    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" event={"ID":"c299cf17-57da-414c-abb1-99d56c5c96a9","Type":"ContainerDied","Data":"5523361980ebabde5198c9f214144f010ac2621f1f7f3f1b0bd4ca5dc80a59fc"}
Oct 31 19:30:14.658389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.658330    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4" event={"ID":"c299cf17-57da-414c-abb1-99d56c5c96a9","Type":"ContainerDied","Data":"dd854ce348964da65d27e4953dc27a8a1f3f76db30dc54792fbfd0cc24b087a6"}
Oct 31 19:30:14.658389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.658350    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd854ce348964da65d27e4953dc27a8a1f3f76db30dc54792fbfd0cc24b087a6"
Oct 31 19:30:14.658389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.658365    5551 scope.go:122] "RemoveContainer" containerID="1bcbfda1df1470784713e52e6212d98f319acada3d9da153f5947429138e3a6d"
Oct 31 19:30:14.667325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.667292    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:30:14.760204 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760178    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-data\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.760341 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760242    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-envoy\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.760414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760363    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-podinfo\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.760506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760439    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-data" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:14.760561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760507    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-certs\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.760623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760608    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-token\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-token\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.760681 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760659    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c299cf17-57da-414c-abb1-99d56c5c96a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c299cf17-57da-414c-abb1-99d56c5c96a9-istiod-ca-cert\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.760737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760721    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-credential-socket\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.760772 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760758    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-socket\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.760881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760857    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-certs" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:14.761007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.760884    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-kube-api-access-rh94x\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-kube-api-access-rh94x\") pod \"c299cf17-57da-414c-abb1-99d56c5c96a9\" (UID: \"c299cf17-57da-414c-abb1-99d56c5c96a9\") "
Oct 31 19:30:14.761078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.761036    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-credential-socket" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:14.761298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.761277    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c299cf17-57da-414c-abb1-99d56c5c96a9-istiod-ca-cert" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:14.761689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.761544    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-socket" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:14.761689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.761566    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c299cf17-57da-414c-abb1-99d56c5c96a9-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.761689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.761593    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.761689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.761607    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.761689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.761620    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.763171 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.763145    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-podinfo" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:30:14.763391 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.763371    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-token" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:14.763646 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.763620    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-envoy" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:14.764126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.764103    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-kube-api-access-rh94x" pod "c299cf17-57da-414c-abb1-99d56c5c96a9" (UID: "c299cf17-57da-414c-abb1-99d56c5c96a9"). InnerVolumeSpecName "kube-api-access-rh94x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:14.863160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.863128    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.863160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.863157    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.863337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.863168    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.863337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.863179    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh94x\" (UniqueName: \"kubernetes.io/projected/c299cf17-57da-414c-abb1-99d56c5c96a9-kube-api-access-rh94x\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.863337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.863190    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c299cf17-57da-414c-abb1-99d56c5c96a9-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:14.909502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:14.902386    5551 scope.go:122] "RemoveContainer" containerID="3c8ddb164529560f084f3db6b7ecfb3c604f73440cc587f045feef5ed6b2cae5"
Oct 31 19:30:15.434015 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.433933    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"]
Oct 31 19:30:15.434621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434605    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c2b2f4f7-7e61-44f9-a974-00de58bb47a9" containerName="payload-processing"
Oct 31 19:30:15.434679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434623    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b2f4f7-7e61-44f9-a974-00de58bb47a9" containerName="payload-processing"
Oct 31 19:30:15.434679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434639    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy"
Oct 31 19:30:15.434679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434646    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy"
Oct 31 19:30:15.434679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434665    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy"
Oct 31 19:30:15.434679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434670    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434683    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434688    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434708    5551 cpu_manager.go:414] "containerMap: removing container" podUID="93e578a0-8c6f-47a0-91b6-7c9309e04d86" containerName="maas-api"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434713    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e578a0-8c6f-47a0-91b6-7c9309e04d86" containerName="maas-api"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434803    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434815    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e578a0-8c6f-47a0-91b6-7c9309e04d86" containerName="maas-api"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434823    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434831    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b2f4f7-7e61-44f9-a974-00de58bb47a9" containerName="payload-processing"
Oct 31 19:30:15.435233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.434838    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" containerName="istio-proxy"
Oct 31 19:30:15.438322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.438306    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.497189 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.497161    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"]
Oct 31 19:30:15.516136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.515991    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:30:15.574161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.574129    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.574400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.574379    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7rr\" (UniqueName: \"kubernetes.io/projected/09f30872-932e-4def-89d1-979e1ab0dfae-kube-api-access-8w7rr\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.574588 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.574568    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.668059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.668020    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" event={"ID":"59e70a9c-2192-4149-af18-38d5885d6f3f","Type":"ContainerStarted","Data":"23fd51380f86b07bb3cbefafc415717f8f1211cdd9034ae30c291e9d3e1cb72b"}
Oct 31 19:30:15.670055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.670026    5551 generic.go:356] "Generic (PLEG): container finished" podID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerID="0a418611fa11c83452969c624b403641240ac687610ccaba91dd800c7634b57f" exitCode=60
Oct 31 19:30:15.670217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.670085    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" event={"ID":"478f858f-3c8c-44df-b97a-3aa8c14a0e86","Type":"ContainerDied","Data":"0a418611fa11c83452969c624b403641240ac687610ccaba91dd800c7634b57f"}
Oct 31 19:30:15.670217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.670152    5551 scope.go:122] "RemoveContainer" containerID="b4c0a8ea4f47d2002422bf41b12819476e93c26ac2a97b6f86f7ed04e0bb5be6"
Oct 31 19:30:15.670585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.670567    5551 scope.go:122] "RemoveContainer" containerID="0a418611fa11c83452969c624b403641240ac687610ccaba91dd800c7634b57f"
Oct 31 19:30:15.670895 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:15.670876    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv_odh-ai-gateway-infra(478f858f-3c8c-44df-b97a-3aa8c14a0e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86"
Oct 31 19:30:15.671736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.671719    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"
Oct 31 19:30:15.675967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.675948    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.676095 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.676079    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7rr\" (UniqueName: \"kubernetes.io/projected/09f30872-932e-4def-89d1-979e1ab0dfae-kube-api-access-8w7rr\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.676221 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.676205    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.676909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.676884    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.678440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.678422    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.732578 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.732549    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7rr\" (UniqueName: \"kubernetes.io/projected/09f30872-932e-4def-89d1-979e1ab0dfae-kube-api-access-8w7rr\") pod \"maas-api-revoke-sub-2e4f5f90-8nhkj\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.752605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.752576    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:15.795375 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.795334    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"]
Oct 31 19:30:15.966424 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.966348    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"]
Oct 31 19:30:15.969788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:15.969761    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-227f0f-openshift-default-6959bf7d8f-455p4"]
Oct 31 19:30:16.179835 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.179810    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"]
Oct 31 19:30:16.181078 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:30:16.181041    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f30872_932e_4def_89d1_979e1ab0dfae.slice/crio-9e83e51bd4cddcdd038598e6b3e9b9a1a9278ed34c9a3b8da95dc15cc48cc95f WatchSource:0}: Error finding container 9e83e51bd4cddcdd038598e6b3e9b9a1a9278ed34c9a3b8da95dc15cc48cc95f: Status 404 returned error can't find the container with id 9e83e51bd4cddcdd038598e6b3e9b9a1a9278ed34c9a3b8da95dc15cc48cc95f
Oct 31 19:30:16.605686 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:16.605594    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:30:16.682775 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.682738    5551 generic.go:356] "Generic (PLEG): container finished" podID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerID="23fd51380f86b07bb3cbefafc415717f8f1211cdd9034ae30c291e9d3e1cb72b" exitCode=60
Oct 31 19:30:16.682973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.682808    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" event={"ID":"59e70a9c-2192-4149-af18-38d5885d6f3f","Type":"ContainerDied","Data":"23fd51380f86b07bb3cbefafc415717f8f1211cdd9034ae30c291e9d3e1cb72b"}
Oct 31 19:30:16.682973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.682863    5551 scope.go:122] "RemoveContainer" containerID="3c8ddb164529560f084f3db6b7ecfb3c604f73440cc587f045feef5ed6b2cae5"
Oct 31 19:30:16.683435 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.683329    5551 scope.go:122] "RemoveContainer" containerID="23fd51380f86b07bb3cbefafc415717f8f1211cdd9034ae30c291e9d3e1cb72b"
Oct 31 19:30:16.683770 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:16.683738    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk_odh-ai-gateway-infra(59e70a9c-2192-4149-af18-38d5885d6f3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f"
Oct 31 19:30:16.687147 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.687125    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj" event={"ID":"09f30872-932e-4def-89d1-979e1ab0dfae","Type":"ContainerStarted","Data":"a0ed80fc7c2dabb0bcb899f6530a62a4bbb7d710ad8d78069abc130d80b3f730"}
Oct 31 19:30:16.687250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.687185    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj" event={"ID":"09f30872-932e-4def-89d1-979e1ab0dfae","Type":"ContainerStarted","Data":"9e83e51bd4cddcdd038598e6b3e9b9a1a9278ed34c9a3b8da95dc15cc48cc95f"}
Oct 31 19:30:16.809733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.809703    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"]
Oct 31 19:30:16.869510 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.869434    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:16.911678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.911650    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c299cf17-57da-414c-abb1-99d56c5c96a9" path="/var/lib/kubelet/pods/c299cf17-57da-414c-abb1-99d56c5c96a9/volumes"
Oct 31 19:30:16.995209 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.995177    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478f858f-3c8c-44df-b97a-3aa8c14a0e86-kube-api-access-nkvh2\" (UniqueName: \"kubernetes.io/projected/478f858f-3c8c-44df-b97a-3aa8c14a0e86-kube-api-access-nkvh2\") pod \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\" (UID: \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\") "
Oct 31 19:30:16.995748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.995284    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478f858f-3c8c-44df-b97a-3aa8c14a0e86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/478f858f-3c8c-44df-b97a-3aa8c14a0e86-maas-api-service-ca\") pod \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\" (UID: \"478f858f-3c8c-44df-b97a-3aa8c14a0e86\") "
Oct 31 19:30:16.995916 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.995889    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478f858f-3c8c-44df-b97a-3aa8c14a0e86-maas-api-service-ca" pod "478f858f-3c8c-44df-b97a-3aa8c14a0e86" (UID: "478f858f-3c8c-44df-b97a-3aa8c14a0e86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:16.996077 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.996057    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/478f858f-3c8c-44df-b97a-3aa8c14a0e86-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:16.997949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:16.997926    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478f858f-3c8c-44df-b97a-3aa8c14a0e86-kube-api-access-nkvh2" pod "478f858f-3c8c-44df-b97a-3aa8c14a0e86" (UID: "478f858f-3c8c-44df-b97a-3aa8c14a0e86"). InnerVolumeSpecName "kube-api-access-nkvh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:17.029355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.029306    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj" podStartSLOduration=2.029292945 podStartE2EDuration="2.029292945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:17.028188847 +0000 UTC m=+4120.259427138" watchObservedRunningTime="2026-10-31 19:30:17.029292945 +0000 UTC m=+4120.260531236"
Oct 31 19:30:17.097245 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.097218    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkvh2\" (UniqueName: \"kubernetes.io/projected/478f858f-3c8c-44df-b97a-3aa8c14a0e86-kube-api-access-nkvh2\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:17.695968 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.695941    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"
Oct 31 19:30:17.696172 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.695970    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv" event={"ID":"478f858f-3c8c-44df-b97a-3aa8c14a0e86","Type":"ContainerDied","Data":"89f3fea6db2333f4366e804c46ed59c589cd78c62ab0fd7500f94e26915ae729"}
Oct 31 19:30:17.696172 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.696045    5551 scope.go:122] "RemoveContainer" containerID="0a418611fa11c83452969c624b403641240ac687610ccaba91dd800c7634b57f"
Oct 31 19:30:17.698059 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.698033    5551 generic.go:356] "Generic (PLEG): container finished" podID="09f30872-932e-4def-89d1-979e1ab0dfae" containerID="a0ed80fc7c2dabb0bcb899f6530a62a4bbb7d710ad8d78069abc130d80b3f730" exitCode=0
Oct 31 19:30:17.698194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.698126    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj" event={"ID":"09f30872-932e-4def-89d1-979e1ab0dfae","Type":"ContainerDied","Data":"a0ed80fc7c2dabb0bcb899f6530a62a4bbb7d710ad8d78069abc130d80b3f730"}
Oct 31 19:30:17.862055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.862028    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"]
Oct 31 19:30:17.865212 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.865188    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:17.869604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.869582    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d8f34e-29891250-nq4hv"]
Oct 31 19:30:17.901724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:17.901700    5551 scope.go:122] "RemoveContainer" containerID="0f20d8144775b146994f11680d85f57e716424ad9dcbf69ebf296bc5b37958c7"
Oct 31 19:30:18.008988 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.008912    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e70a9c-2192-4149-af18-38d5885d6f3f-kube-api-access-txp5k\" (UniqueName: \"kubernetes.io/projected/59e70a9c-2192-4149-af18-38d5885d6f3f-kube-api-access-txp5k\") pod \"59e70a9c-2192-4149-af18-38d5885d6f3f\" (UID: \"59e70a9c-2192-4149-af18-38d5885d6f3f\") "
Oct 31 19:30:18.009345 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.009119    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e70a9c-2192-4149-af18-38d5885d6f3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e70a9c-2192-4149-af18-38d5885d6f3f-maas-api-service-ca\") pod \"59e70a9c-2192-4149-af18-38d5885d6f3f\" (UID: \"59e70a9c-2192-4149-af18-38d5885d6f3f\") "
Oct 31 19:30:18.009472 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.009438    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e70a9c-2192-4149-af18-38d5885d6f3f-maas-api-service-ca" pod "59e70a9c-2192-4149-af18-38d5885d6f3f" (UID: "59e70a9c-2192-4149-af18-38d5885d6f3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:18.009748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.009730    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e70a9c-2192-4149-af18-38d5885d6f3f-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:18.011201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.011180    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e70a9c-2192-4149-af18-38d5885d6f3f-kube-api-access-txp5k" pod "59e70a9c-2192-4149-af18-38d5885d6f3f" (UID: "59e70a9c-2192-4149-af18-38d5885d6f3f"). InnerVolumeSpecName "kube-api-access-txp5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:18.110568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.110537    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txp5k\" (UniqueName: \"kubernetes.io/projected/59e70a9c-2192-4149-af18-38d5885d6f3f-kube-api-access-txp5k\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:18.297372 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.297285    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"]
Oct 31 19:30:18.297582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.297550    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s" podUID="2cf3395c-b065-43e2-af60-9ce6ee366441" containerName="maas-api" containerID="cri-o://247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7" gracePeriod=30
Oct 31 19:30:18.596781 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.596756    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:30:18.727484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.725268    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf3395c-b065-43e2-af60-9ce6ee366441-kube-api-access-kn6wc\" (UniqueName: \"kubernetes.io/projected/2cf3395c-b065-43e2-af60-9ce6ee366441-kube-api-access-kn6wc\") pod \"2cf3395c-b065-43e2-af60-9ce6ee366441\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") "
Oct 31 19:30:18.727484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.725404    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls\") pod \"2cf3395c-b065-43e2-af60-9ce6ee366441\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") "
Oct 31 19:30:18.727484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.725496    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf3395c-b065-43e2-af60-9ce6ee366441-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3395c-b065-43e2-af60-9ce6ee366441-openshift-service-ca\") pod \"2cf3395c-b065-43e2-af60-9ce6ee366441\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") "
Oct 31 19:30:18.727484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.725649    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-metrics-certs\") pod \"2cf3395c-b065-43e2-af60-9ce6ee366441\" (UID: \"2cf3395c-b065-43e2-af60-9ce6ee366441\") "
Oct 31 19:30:18.735066 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.735012    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf3395c-b065-43e2-af60-9ce6ee366441-openshift-service-ca" pod "2cf3395c-b065-43e2-af60-9ce6ee366441" (UID: "2cf3395c-b065-43e2-af60-9ce6ee366441"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:18.736781 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.736078    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3395c-b065-43e2-af60-9ce6ee366441-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:18.738337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.738273    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-metrics-certs" pod "2cf3395c-b065-43e2-af60-9ce6ee366441" (UID: "2cf3395c-b065-43e2-af60-9ce6ee366441"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:30:18.742165 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.740922    5551 generic.go:356] "Generic (PLEG): container finished" podID="2cf3395c-b065-43e2-af60-9ce6ee366441" containerID="247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7" exitCode=0
Oct 31 19:30:18.742165 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.741071    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s" event={"ID":"2cf3395c-b065-43e2-af60-9ce6ee366441","Type":"ContainerDied","Data":"247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7"}
Oct 31 19:30:18.742165 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.741121    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s" event={"ID":"2cf3395c-b065-43e2-af60-9ce6ee366441","Type":"ContainerDied","Data":"ad00ea2b206f3a16a1dbcd210d954999ff5d31d98efe7e11423532d606b609fd"}
Oct 31 19:30:18.742165 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.741149    5551 scope.go:122] "RemoveContainer" containerID="247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7"
Oct 31 19:30:18.742165 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.741263    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"
Oct 31 19:30:18.746509 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.746136    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk" event={"ID":"59e70a9c-2192-4149-af18-38d5885d6f3f","Type":"ContainerDied","Data":"1efed8a6c3c24724a49b698e95035f2b1c37b6b4f7188e74a496b2f459b17089"}
Oct 31 19:30:18.746509 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.746259    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"
Oct 31 19:30:18.752482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.752107    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" event={"ID":"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b","Type":"ContainerStarted","Data":"b1534b9c239a3b9338c12fcc05c03807a4e3678cc5f8fc8d442ba55514108133"}
Oct 31 19:30:18.775160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.775118    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls" pod "2cf3395c-b065-43e2-af60-9ce6ee366441" (UID: "2cf3395c-b065-43e2-af60-9ce6ee366441"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:30:18.778991 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.778953    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf3395c-b065-43e2-af60-9ce6ee366441-kube-api-access-kn6wc" pod "2cf3395c-b065-43e2-af60-9ce6ee366441" (UID: "2cf3395c-b065-43e2-af60-9ce6ee366441"). InnerVolumeSpecName "kube-api-access-kn6wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:18.785418 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.785384    5551 scope.go:122] "RemoveContainer" containerID="247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7"
Oct 31 19:30:18.785821 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:18.785799    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7\": container with ID starting with 247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7 not found: ID does not exist" containerID="247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7"
Oct 31 19:30:18.785927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.785835    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7"} err="failed to get container status \"247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7\": rpc error: code = NotFound desc = could not find container \"247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7\": container with ID starting with 247e9432162679e4aea013ef64b3c7daef61d30ae864fb51a917668ede1cdef7 not found: ID does not exist"
Oct 31 19:30:18.785927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.785863    5551 scope.go:122] "RemoveContainer" containerID="23fd51380f86b07bb3cbefafc415717f8f1211cdd9034ae30c291e9d3e1cb72b"
Oct 31 19:30:18.841603 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.841568    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:18.841755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.841609    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn6wc\" (UniqueName: \"kubernetes.io/projected/2cf3395c-b065-43e2-af60-9ce6ee366441-kube-api-access-kn6wc\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:18.841755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.841628    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3395c-b065-43e2-af60-9ce6ee366441-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:18.902756 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.902314    5551 scope.go:122] "RemoveContainer" containerID="fdb3e6d46224e2ecaf1c0d9f78d1bd214fb95c871d409ec4196615778699769b"
Oct 31 19:30:18.929014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.928980    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" path="/var/lib/kubelet/pods/478f858f-3c8c-44df-b97a-3aa8c14a0e86/volumes"
Oct 31 19:30:18.938753 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:18.938728    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:19.024952 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.024921    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"]
Oct 31 19:30:19.030147 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.030114    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c97ce8-29891250-k2msk"]
Oct 31 19:30:19.044041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.044012    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-xks-ca\") pod \"09f30872-932e-4def-89d1-979e1ab0dfae\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") "
Oct 31 19:30:19.044221 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.044193    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f30872-932e-4def-89d1-979e1ab0dfae-kube-api-access-8w7rr\" (UniqueName: \"kubernetes.io/projected/09f30872-932e-4def-89d1-979e1ab0dfae-kube-api-access-8w7rr\") pod \"09f30872-932e-4def-89d1-979e1ab0dfae\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") "
Oct 31 19:30:19.044281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.044241    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-openshift-ca\") pod \"09f30872-932e-4def-89d1-979e1ab0dfae\" (UID: \"09f30872-932e-4def-89d1-979e1ab0dfae\") "
Oct 31 19:30:19.044698 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.044667    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-openshift-ca" pod "09f30872-932e-4def-89d1-979e1ab0dfae" (UID: "09f30872-932e-4def-89d1-979e1ab0dfae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:19.045181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.045156    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:19.047171 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.047146    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-xks-ca" pod "09f30872-932e-4def-89d1-979e1ab0dfae" (UID: "09f30872-932e-4def-89d1-979e1ab0dfae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:30:19.047264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.047182    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f30872-932e-4def-89d1-979e1ab0dfae-kube-api-access-8w7rr" pod "09f30872-932e-4def-89d1-979e1ab0dfae" (UID: "09f30872-932e-4def-89d1-979e1ab0dfae"). InnerVolumeSpecName "kube-api-access-8w7rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:19.147518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.147475    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f30872-932e-4def-89d1-979e1ab0dfae-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:19.147518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.147515    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w7rr\" (UniqueName: \"kubernetes.io/projected/09f30872-932e-4def-89d1-979e1ab0dfae-kube-api-access-8w7rr\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:19.248731 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.248562    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"]
Oct 31 19:30:19.252832 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.252807    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b70248-778c84f566-vh86s"]
Oct 31 19:30:19.770199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.770153    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" event={"ID":"6dd7ec9a-60e8-4553-afb6-aac6e383a844","Type":"ContainerStarted","Data":"b6de8e62cce968e60000f822d2f0c45c6aa5a32f136cf2466cf2dc0334834b58"}
Oct 31 19:30:19.774891 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.774862    5551 generic.go:356] "Generic (PLEG): container finished" podID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerID="b1534b9c239a3b9338c12fcc05c03807a4e3678cc5f8fc8d442ba55514108133" exitCode=60
Oct 31 19:30:19.775037 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.774943    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" event={"ID":"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b","Type":"ContainerDied","Data":"b1534b9c239a3b9338c12fcc05c03807a4e3678cc5f8fc8d442ba55514108133"}
Oct 31 19:30:19.775037 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.775007    5551 scope.go:122] "RemoveContainer" containerID="0f20d8144775b146994f11680d85f57e716424ad9dcbf69ebf296bc5b37958c7"
Oct 31 19:30:19.775335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.775316    5551 scope.go:122] "RemoveContainer" containerID="b1534b9c239a3b9338c12fcc05c03807a4e3678cc5f8fc8d442ba55514108133"
Oct 31 19:30:19.775682 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:19.775659    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb_odh-ai-gateway-infra(c7c3782c-02a6-4fda-8fdc-0d423cb9f39b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b"
Oct 31 19:30:19.776970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.776938    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"
Oct 31 19:30:19.777078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.776973    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj" event={"ID":"09f30872-932e-4def-89d1-979e1ab0dfae","Type":"ContainerDied","Data":"9e83e51bd4cddcdd038598e6b3e9b9a1a9278ed34c9a3b8da95dc15cc48cc95f"}
Oct 31 19:30:19.777078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.777015    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e83e51bd4cddcdd038598e6b3e9b9a1a9278ed34c9a3b8da95dc15cc48cc95f"
Oct 31 19:30:19.923038 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:19.923000    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"]
Oct 31 19:30:20.787685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.787654    5551 generic.go:356] "Generic (PLEG): container finished" podID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerID="b6de8e62cce968e60000f822d2f0c45c6aa5a32f136cf2466cf2dc0334834b58" exitCode=60
Oct 31 19:30:20.788177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.787700    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" event={"ID":"6dd7ec9a-60e8-4553-afb6-aac6e383a844","Type":"ContainerDied","Data":"b6de8e62cce968e60000f822d2f0c45c6aa5a32f136cf2466cf2dc0334834b58"}
Oct 31 19:30:20.788177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.787753    5551 scope.go:122] "RemoveContainer" containerID="fdb3e6d46224e2ecaf1c0d9f78d1bd214fb95c871d409ec4196615778699769b"
Oct 31 19:30:20.788177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.788062    5551 scope.go:122] "RemoveContainer" containerID="b6de8e62cce968e60000f822d2f0c45c6aa5a32f136cf2466cf2dc0334834b58"
Oct 31 19:30:20.788377 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:20.788331    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7_odh-ai-gateway-infra(6dd7ec9a-60e8-4553-afb6-aac6e383a844)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844"
Oct 31 19:30:20.918925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.918889    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf3395c-b065-43e2-af60-9ce6ee366441" path="/var/lib/kubelet/pods/2cf3395c-b065-43e2-af60-9ce6ee366441/volumes"
Oct 31 19:30:20.919572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.919548    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" path="/var/lib/kubelet/pods/59e70a9c-2192-4149-af18-38d5885d6f3f/volumes"
Oct 31 19:30:20.920031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.919989    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"]
Oct 31 19:30:20.958907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.958876    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:20.981334 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.981301    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-kube-api-access-gxjxv\" (UniqueName: \"kubernetes.io/projected/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-kube-api-access-gxjxv\") pod \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\" (UID: \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\") "
Oct 31 19:30:20.981534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.981502    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-maas-api-service-ca\") pod \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\" (UID: \"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b\") "
Oct 31 19:30:20.981915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.981885    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-maas-api-service-ca" pod "c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" (UID: "c7c3782c-02a6-4fda-8fdc-0d423cb9f39b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:20.982754 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.982291    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:20.984679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:20.984654    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-kube-api-access-gxjxv" pod "c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" (UID: "c7c3782c-02a6-4fda-8fdc-0d423cb9f39b"). InnerVolumeSpecName "kube-api-access-gxjxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:21.084172 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.084088    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxjxv\" (UniqueName: \"kubernetes.io/projected/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b-kube-api-access-gxjxv\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:21.799066 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.799043    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"
Oct 31 19:30:21.799478 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.799095    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb" event={"ID":"c7c3782c-02a6-4fda-8fdc-0d423cb9f39b","Type":"ContainerDied","Data":"abb62f4b318408ce97089c946ac4f3db2ab9e6ccf0e971bf1fb37b255e072f34"}
Oct 31 19:30:21.799478 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.799165    5551 scope.go:122] "RemoveContainer" containerID="b1534b9c239a3b9338c12fcc05c03807a4e3678cc5f8fc8d442ba55514108133"
Oct 31 19:30:21.966674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.966651    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:21.996099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.996068    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd7ec9a-60e8-4553-afb6-aac6e383a844-kube-api-access-cxzkv\" (UniqueName: \"kubernetes.io/projected/6dd7ec9a-60e8-4553-afb6-aac6e383a844-kube-api-access-cxzkv\") pod \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\" (UID: \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\") "
Oct 31 19:30:21.996267 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.996193    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd7ec9a-60e8-4553-afb6-aac6e383a844-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd7ec9a-60e8-4553-afb6-aac6e383a844-maas-api-service-ca\") pod \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\" (UID: \"6dd7ec9a-60e8-4553-afb6-aac6e383a844\") "
Oct 31 19:30:21.996590 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.996564    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd7ec9a-60e8-4553-afb6-aac6e383a844-maas-api-service-ca" pod "6dd7ec9a-60e8-4553-afb6-aac6e383a844" (UID: "6dd7ec9a-60e8-4553-afb6-aac6e383a844"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:21.996847 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.996827    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd7ec9a-60e8-4553-afb6-aac6e383a844-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:21.998925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:21.998891    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd7ec9a-60e8-4553-afb6-aac6e383a844-kube-api-access-cxzkv" pod "6dd7ec9a-60e8-4553-afb6-aac6e383a844" (UID: "6dd7ec9a-60e8-4553-afb6-aac6e383a844"). InnerVolumeSpecName "kube-api-access-cxzkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:22.021801 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.021772    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"]
Oct 31 19:30:22.024940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.024919    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bbd8c6-29891250-km9kb"]
Oct 31 19:30:22.097966 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.097888    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzkv\" (UniqueName: \"kubernetes.io/projected/6dd7ec9a-60e8-4553-afb6-aac6e383a844-kube-api-access-cxzkv\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:22.808049 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.808012    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"
Oct 31 19:30:22.808545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.808050    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7" event={"ID":"6dd7ec9a-60e8-4553-afb6-aac6e383a844","Type":"ContainerDied","Data":"fb97c8fb7b2a8bb919ee6ae1350c40ce3ec1c56f50bfd9affbe525c25b54ab8a"}
Oct 31 19:30:22.808545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.808113    5551 scope.go:122] "RemoveContainer" containerID="b6de8e62cce968e60000f822d2f0c45c6aa5a32f136cf2466cf2dc0334834b58"
Oct 31 19:30:22.966147 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.962646    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" path="/var/lib/kubelet/pods/c7c3782c-02a6-4fda-8fdc-0d423cb9f39b/volumes"
Oct 31 19:30:22.985879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.985848    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"]
Oct 31 19:30:22.991333 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:22.991299    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891250-5k2n7"]
Oct 31 19:30:24.909831 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:24.909801    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" path="/var/lib/kubelet/pods/6dd7ec9a-60e8-4553-afb6-aac6e383a844/volumes"
Oct 31 19:30:28.094693 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.094645    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"]
Oct 31 19:30:28.095257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095243    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.095257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095257    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095270    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095276    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095297    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095302    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095317    5551 cpu_manager.go:414] "containerMap: removing container" podUID="2cf3395c-b065-43e2-af60-9ce6ee366441" containerName="maas-api"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095323    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf3395c-b065-43e2-af60-9ce6ee366441" containerName="maas-api"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095330    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095334    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095343    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095347    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095362    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095367    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095375    5551 cpu_manager.go:414] "containerMap: removing container" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095379    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095387    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.095381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095391    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095399    5551 cpu_manager.go:414] "containerMap: removing container" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095403    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095410    5551 cpu_manager.go:414] "containerMap: removing container" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095414    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095422    5551 cpu_manager.go:414] "containerMap: removing container" podUID="09f30872-932e-4def-89d1-979e1ab0dfae" containerName="revoke-keys"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095426    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f30872-932e-4def-89d1-979e1ab0dfae" containerName="revoke-keys"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095440    5551 cpu_manager.go:414] "containerMap: removing container" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095444    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095544    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095551    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f30872-932e-4def-89d1-979e1ab0dfae" containerName="revoke-keys"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095560    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095566    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095573    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095579    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095585    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf3395c-b065-43e2-af60-9ce6ee366441" containerName="maas-api"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095590    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd7ec9a-60e8-4553-afb6-aac6e383a844" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095595    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095600    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9ac8e8-f19e-4ccb-a5f5-e08cfb3c10b4" containerName="istio-proxy"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095604    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095610    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:30:28.096250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.095615    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c3782c-02a6-4fda-8fdc-0d423cb9f39b" containerName="cleanup"
Oct 31 19:30:28.104571 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.104529    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.211357 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.211322    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"]
Oct 31 19:30:28.217106 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217080    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d227addd-3b02-4f7d-aed0-a681fd8d680d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.217243 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217118    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.217295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217277    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2plxj\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-kube-api-access-2plxj\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.217390 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217377    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.217446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217412    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.217446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217435    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.217585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217484    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.217585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217573    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.217692 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.217675    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318413    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2plxj\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-kube-api-access-2plxj\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318525    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318552    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318574    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318590    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318611    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318656    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318701    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d227addd-3b02-4f7d-aed0-a681fd8d680d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.318909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.318737    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.319112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.319089    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.319182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.319129    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.319227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.319184    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.319267 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.319218    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.319829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.319801    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d227addd-3b02-4f7d-aed0-a681fd8d680d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.320924 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.320905    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.321231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.321211    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.375930 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.375896    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.376124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.376101    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2plxj\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-kube-api-access-2plxj\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9ckv2\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.419983 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.419946    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:28.860254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.860204    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"]
Oct 31 19:30:28.862397 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:30:28.862366    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd227addd_3b02_4f7d_aed0_a681fd8d680d.slice/crio-a671ac2f6d8496abe3d64d6d18d7551aed56b2771fa81d9c1ba85df39c2dd4be WatchSource:0}: Error finding container a671ac2f6d8496abe3d64d6d18d7551aed56b2771fa81d9c1ba85df39c2dd4be: Status 404 returned error can't find the container with id a671ac2f6d8496abe3d64d6d18d7551aed56b2771fa81d9c1ba85df39c2dd4be
Oct 31 19:30:28.865592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.865554    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:30:28.865688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.865645    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:30:28.865748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:28.865688    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:30:29.898374 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:29.898325    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" event={"ID":"d227addd-3b02-4f7d-aed0-a681fd8d680d","Type":"ContainerStarted","Data":"f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5"}
Oct 31 19:30:29.898879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:29.898394    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" event={"ID":"d227addd-3b02-4f7d-aed0-a681fd8d680d","Type":"ContainerStarted","Data":"a671ac2f6d8496abe3d64d6d18d7551aed56b2771fa81d9c1ba85df39c2dd4be"}
Oct 31 19:30:30.026530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:30.026474    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podStartSLOduration=2.026443877 podStartE2EDuration="2.026443877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:30.023985583 +0000 UTC m=+4133.255223871" watchObservedRunningTime="2026-10-31 19:30:30.026443877 +0000 UTC m=+4133.257682166"
Oct 31 19:30:30.421169 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:30.421131    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:30.422852 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:30.422824    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9ckv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 19:30:30.423010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:30.422875    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 19:30:31.420725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:31.420691    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9ckv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 19:30:31.421141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:31.420742    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 19:30:32.421334 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:32.421289    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9ckv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 19:30:32.421795 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:32.421367    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 19:30:33.420562 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:33.420522    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9ckv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 19:30:33.420759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:33.420601    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 19:30:34.421416 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:34.421368    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9ckv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 19:30:34.421855 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:34.421483    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 19:30:35.423485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:35.423364    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9ckv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 19:30:35.423485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:35.423441    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 19:30:36.428022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:36.427971    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9ckv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 19:30:36.428726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:36.428689    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 19:30:36.450657 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:36.450619    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:30:36.450927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:36.450900    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:30:36.451112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:36.451096    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hzvrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:30:36.451264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:36.451246    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:30:37.420319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:37.420279    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9ckv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 19:30:37.420553 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:37.420345    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 19:30:38.662361 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:38.662326    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:38.662870 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:38.662792    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:38.663628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:38.663606    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:30:43.862027 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:43.861895    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"]
Oct 31 19:30:43.862686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:43.862311    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy" containerID="cri-o://cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4" gracePeriod=30
Oct 31 19:30:45.537927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:45.537893    5551 patch_prober.go:29] interesting pod/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:30:45.538407 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:45.537968    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:30:46.605338 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:46.605301    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:30:49.135952 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.135896    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:30:49.254903 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.254863    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-credential-socket\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.255126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.254950    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-socket\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.255126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255036    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-podinfo\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.255126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255066    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-data\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.255126 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255106    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-envoy\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.255344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255203    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-credential-socket" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:49.255572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255291    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-certs\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.255725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255593    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istiod-ca-cert\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.255725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255299    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-socket" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:49.255725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255365    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-data" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:49.255725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255551    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-certs" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:49.255725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255630    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-token\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-token\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.256201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.255724    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-kube-api-access-htshj\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-kube-api-access-htshj\") pod \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\" (UID: \"e4dd413c-bbce-4a1b-9068-d8bb52df6701\") "
Oct 31 19:30:49.256274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.256240    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istiod-ca-cert" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:49.256746 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.256720    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.256882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.256748    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.256882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.256763    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.256882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.256777    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.256882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.256792    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.258083 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.258056    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-envoy" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:30:49.258204 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.258120    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-podinfo" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:30:49.258388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.258364    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-token" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:49.258593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.258572    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-kube-api-access-htshj" pod "e4dd413c-bbce-4a1b-9068-d8bb52df6701" (UID: "e4dd413c-bbce-4a1b-9068-d8bb52df6701"). InnerVolumeSpecName "kube-api-access-htshj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:49.358369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.358330    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.358369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.358364    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.358369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.358374    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.358650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.358385    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htshj\" (UniqueName: \"kubernetes.io/projected/e4dd413c-bbce-4a1b-9068-d8bb52df6701-kube-api-access-htshj\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:49.446747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.446665    5551 generic.go:356] "Generic (PLEG): container finished" podID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerID="cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4" exitCode=0
Oct 31 19:30:49.446747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.446723    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"
Oct 31 19:30:49.446951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.446746    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" event={"ID":"e4dd413c-bbce-4a1b-9068-d8bb52df6701","Type":"ContainerDied","Data":"cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4"}
Oct 31 19:30:49.446951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.446807    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6" event={"ID":"e4dd413c-bbce-4a1b-9068-d8bb52df6701","Type":"ContainerDied","Data":"61255991c2f1b25e3daf96f9513426f6401b0d9575fa7d7188ce8bf983bbfdf6"}
Oct 31 19:30:49.446951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.446832    5551 scope.go:122] "RemoveContainer" containerID="cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4"
Oct 31 19:30:49.458908 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.458887    5551 scope.go:122] "RemoveContainer" containerID="44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb"
Oct 31 19:30:49.473779 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.473676    5551 scope.go:122] "RemoveContainer" containerID="cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4"
Oct 31 19:30:49.474060 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:49.474029    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4\": container with ID starting with cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4 not found: ID does not exist" containerID="cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4"
Oct 31 19:30:49.474170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.474071    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4"} err="failed to get container status \"cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4\": rpc error: code = NotFound desc = could not find container \"cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4\": container with ID starting with cf5d95348b76ed09b2d1bfacbca894d945319071b28ff43c9e8b86313bf49ad4 not found: ID does not exist"
Oct 31 19:30:49.474170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.474097    5551 scope.go:122] "RemoveContainer" containerID="44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb"
Oct 31 19:30:49.474884 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:30:49.474854    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb\": container with ID starting with 44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb not found: ID does not exist" containerID="44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb"
Oct 31 19:30:49.474985 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.474898    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb"} err="failed to get container status \"44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb\": rpc error: code = NotFound desc = could not find container \"44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb\": container with ID starting with 44efc184c202d4dd1d893d980d285a1bc1e0d81922871147d46afa61eaa862fb not found: ID does not exist"
Oct 31 19:30:49.613985 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.613953    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"]
Oct 31 19:30:49.620156 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:49.620131    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c249cd-openshift-default-79cd855d9c-sz4n6"]
Oct 31 19:30:50.909416 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:30:50.909375    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" path="/var/lib/kubelet/pods/e4dd413c-bbce-4a1b-9068-d8bb52df6701/volumes"
Oct 31 19:31:16.604399 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:31:16.604360    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:31:40.554417 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:31:40.554391    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:31:40.556367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:31:40.556345    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:31:46.605126 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:31:46.605078    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:32:02.599973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.599934    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"]
Oct 31 19:32:02.600661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.600176    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5" podUID="574f9e7c-0781-451a-a0e0-e7800f36f02d" containerName="maas-api" containerID="cri-o://315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe" gracePeriod=30
Oct 31 19:32:02.890764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.890739    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:32:02.976354 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.976322    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/574f9e7c-0781-451a-a0e0-e7800f36f02d-kube-api-access-c92jf\" (UniqueName: \"kubernetes.io/projected/574f9e7c-0781-451a-a0e0-e7800f36f02d-kube-api-access-c92jf\") pod \"574f9e7c-0781-451a-a0e0-e7800f36f02d\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") "
Oct 31 19:32:02.976534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.976383    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls\") pod \"574f9e7c-0781-451a-a0e0-e7800f36f02d\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") "
Oct 31 19:32:02.976623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.976611    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/574f9e7c-0781-451a-a0e0-e7800f36f02d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/574f9e7c-0781-451a-a0e0-e7800f36f02d-openshift-service-ca\") pod \"574f9e7c-0781-451a-a0e0-e7800f36f02d\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") "
Oct 31 19:32:02.976668 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.976637    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-metrics-certs\") pod \"574f9e7c-0781-451a-a0e0-e7800f36f02d\" (UID: \"574f9e7c-0781-451a-a0e0-e7800f36f02d\") "
Oct 31 19:32:02.976953 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.976925    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/574f9e7c-0781-451a-a0e0-e7800f36f02d-openshift-service-ca" pod "574f9e7c-0781-451a-a0e0-e7800f36f02d" (UID: "574f9e7c-0781-451a-a0e0-e7800f36f02d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:32:02.977227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.977207    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/574f9e7c-0781-451a-a0e0-e7800f36f02d-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:02.979115 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.979087    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-metrics-certs" pod "574f9e7c-0781-451a-a0e0-e7800f36f02d" (UID: "574f9e7c-0781-451a-a0e0-e7800f36f02d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:32:02.979236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.979168    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls" pod "574f9e7c-0781-451a-a0e0-e7800f36f02d" (UID: "574f9e7c-0781-451a-a0e0-e7800f36f02d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:32:02.979236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:02.979212    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574f9e7c-0781-451a-a0e0-e7800f36f02d-kube-api-access-c92jf" pod "574f9e7c-0781-451a-a0e0-e7800f36f02d" (UID: "574f9e7c-0781-451a-a0e0-e7800f36f02d"). InnerVolumeSpecName "kube-api-access-c92jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:32:03.076854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.076810    5551 generic.go:356] "Generic (PLEG): container finished" podID="574f9e7c-0781-451a-a0e0-e7800f36f02d" containerID="315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe" exitCode=0
Oct 31 19:32:03.077074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.076879    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"
Oct 31 19:32:03.077074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.076911    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5" event={"ID":"574f9e7c-0781-451a-a0e0-e7800f36f02d","Type":"ContainerDied","Data":"315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe"}
Oct 31 19:32:03.077074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.076987    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5" event={"ID":"574f9e7c-0781-451a-a0e0-e7800f36f02d","Type":"ContainerDied","Data":"a92424d8fc8d4d0659d040f9340de89f0c8dd02cddc688a47c51e8e120275400"}
Oct 31 19:32:03.077074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.077035    5551 scope.go:122] "RemoveContainer" containerID="315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe"
Oct 31 19:32:03.078248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.078227    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:03.078355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.078252    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92jf\" (UniqueName: \"kubernetes.io/projected/574f9e7c-0781-451a-a0e0-e7800f36f02d-kube-api-access-c92jf\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:03.078355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.078268    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/574f9e7c-0781-451a-a0e0-e7800f36f02d-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:03.098587 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.098559    5551 scope.go:122] "RemoveContainer" containerID="315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe"
Oct 31 19:32:03.098956 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:32:03.098928    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe\": container with ID starting with 315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe not found: ID does not exist" containerID="315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe"
Oct 31 19:32:03.099041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.098971    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe"} err="failed to get container status \"315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe\": rpc error: code = NotFound desc = could not find container \"315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe\": container with ID starting with 315b44de657122b9550d102e8f04e66059490427effc02f52f755a50232d53fe not found: ID does not exist"
Oct 31 19:32:03.243702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.243671    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"]
Oct 31 19:32:03.247625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:03.247599    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97ce8-7df46f6698-jt5w5"]
Oct 31 19:32:04.909821 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:04.909784    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="574f9e7c-0781-451a-a0e0-e7800f36f02d" path="/var/lib/kubelet/pods/574f9e7c-0781-451a-a0e0-e7800f36f02d/volumes"
Oct 31 19:32:06.578134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:06.578103    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"]
Oct 31 19:32:06.578653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:06.578382    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy" containerID="cri-o://e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d" gracePeriod=30
Oct 31 19:32:11.832749 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.832726    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:32:11.897091 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897063    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-podinfo\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897134    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-socket\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897162    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-data\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897208    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-token\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-token\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897296    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istiod-ca-cert\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897603 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897356    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-credential-socket\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897603 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897415    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-envoy\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897603 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897509    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-kube-api-access-f8696\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-kube-api-access-f8696\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897603 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897532    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-socket" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:32:11.897833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897611    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-certs\") pod \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\" (UID: \"8fa859b3-2ed2-4a73-bfa6-51028ab181b1\") "
Oct 31 19:32:11.897833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.897768    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istiod-ca-cert" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:32:11.898089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.898006    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-credential-socket" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:32:11.898089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.898054    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-data" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:32:11.898089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.898072    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-certs" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:32:11.898594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.898419    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:11.898594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.898445    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:11.898594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.898476    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:11.898594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.898492    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:11.898594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.898506    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:11.899925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.899902    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-envoy" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:32:11.899925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.899904    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-token" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:32:11.900092 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.900017    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-podinfo" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:32:11.900269 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:11.900252    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-kube-api-access-f8696" pod "8fa859b3-2ed2-4a73-bfa6-51028ab181b1" (UID: "8fa859b3-2ed2-4a73-bfa6-51028ab181b1"). InnerVolumeSpecName "kube-api-access-f8696". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:32:12.001611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.001578    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:12.001611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.001614    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8696\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-kube-api-access-f8696\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:12.001848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.001630    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:12.001848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.001645    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa859b3-2ed2-4a73-bfa6-51028ab181b1-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:12.144925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.144891    5551 generic.go:356] "Generic (PLEG): container finished" podID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerID="e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d" exitCode=0
Oct 31 19:32:12.145121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.144941    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"
Oct 31 19:32:12.145121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.144982    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" event={"ID":"8fa859b3-2ed2-4a73-bfa6-51028ab181b1","Type":"ContainerDied","Data":"e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d"}
Oct 31 19:32:12.145121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.145031    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz" event={"ID":"8fa859b3-2ed2-4a73-bfa6-51028ab181b1","Type":"ContainerDied","Data":"559e926fe7ae22b76db3538489d15f5e42e7c4c78a93d19fe05d9332e82988ee"}
Oct 31 19:32:12.145121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.145052    5551 scope.go:122] "RemoveContainer" containerID="e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d"
Oct 31 19:32:12.156768 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.156751    5551 scope.go:122] "RemoveContainer" containerID="21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c"
Oct 31 19:32:12.166521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.166503    5551 scope.go:122] "RemoveContainer" containerID="e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d"
Oct 31 19:32:12.166818 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:32:12.166795    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d\": container with ID starting with e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d not found: ID does not exist" containerID="e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d"
Oct 31 19:32:12.166886 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.166831    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d"} err="failed to get container status \"e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d\": rpc error: code = NotFound desc = could not find container \"e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d\": container with ID starting with e16d009503af7d6c91413482055507aa5c9753a40de521c174000f2d5282402d not found: ID does not exist"
Oct 31 19:32:12.166886 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.166853    5551 scope.go:122] "RemoveContainer" containerID="21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c"
Oct 31 19:32:12.167127 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:32:12.167103    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c\": container with ID starting with 21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c not found: ID does not exist" containerID="21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c"
Oct 31 19:32:12.167179 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.167138    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c"} err="failed to get container status \"21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c\": rpc error: code = NotFound desc = could not find container \"21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c\": container with ID starting with 21ddce8c2d5efb2b7b868bdc98d23c87ede7347713a37a8aa3db1e4af94b003c not found: ID does not exist"
Oct 31 19:32:12.309618 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.309582    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"]
Oct 31 19:32:12.314259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.314231    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c97ce8-openshift-default-5c56fd6669-qqcqz"]
Oct 31 19:32:12.908968 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:32:12.908936    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" path="/var/lib/kubelet/pods/8fa859b3-2ed2-4a73-bfa6-51028ab181b1/volumes"
Oct 31 19:32:16.606852 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:32:16.606820    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:32:46.604914 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:32:46.604875    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:33:16.606216 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:33:16.606163    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:33:30.292925 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:30.292890    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"]
Oct 31 19:33:30.293367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:30.293116    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy" containerID="cri-o://bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1" gracePeriod=30
Oct 31 19:33:34.398320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398289    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"]
Oct 31 19:33:34.398892 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398877    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy"
Oct 31 19:33:34.398943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398894    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy"
Oct 31 19:33:34.398943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398935    5551 cpu_manager.go:414] "containerMap: removing container" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:33:34.398943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398941    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:33:34.399046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398956    5551 cpu_manager.go:414] "containerMap: removing container" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:33:34.399046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398961    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:33:34.399046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398971    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy"
Oct 31 19:33:34.399046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398975    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy"
Oct 31 19:33:34.399046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398991    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy"
Oct 31 19:33:34.399046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.398996    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy"
Oct 31 19:33:34.399046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399008    5551 cpu_manager.go:414] "containerMap: removing container" podUID="574f9e7c-0781-451a-a0e0-e7800f36f02d" containerName="maas-api"
Oct 31 19:33:34.399046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399014    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574f9e7c-0781-451a-a0e0-e7800f36f02d" containerName="maas-api"
Oct 31 19:33:34.399364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399090    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="478f858f-3c8c-44df-b97a-3aa8c14a0e86" containerName="cleanup"
Oct 31 19:33:34.399364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399100    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e70a9c-2192-4149-af18-38d5885d6f3f" containerName="cleanup"
Oct 31 19:33:34.399364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399108    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy"
Oct 31 19:33:34.399364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399116    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy"
Oct 31 19:33:34.399364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399123    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4dd413c-bbce-4a1b-9068-d8bb52df6701" containerName="istio-proxy"
Oct 31 19:33:34.399364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399130    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="574f9e7c-0781-451a-a0e0-e7800f36f02d" containerName="maas-api"
Oct 31 19:33:34.399364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.399137    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy"
Oct 31 19:33:34.402600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.402582    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.433426 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.433387    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b55310-85c9-403e-98de-4be997a7aeaf-tls-certs\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.433557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.433439    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-home\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.433557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.433544    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmrl\" (UniqueName: \"kubernetes.io/projected/88b55310-85c9-403e-98de-4be997a7aeaf-kube-api-access-qmmrl\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.433658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.433646    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-model-cache\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.433815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.433794    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-tmp-dir\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.433856 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.433827    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-dshm\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.505319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.505288    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-dbceb9\"/\"kube-root-ca.crt\""
Oct 31 19:33:34.505319 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.505288    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-dbceb9\"/\"openshift-service-ca.crt\""
Oct 31 19:33:34.505582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.505331    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-dbceb9\"/\"sub-test-model-dbceb9-kserve-self-signed-certs\""
Oct 31 19:33:34.505582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.505390    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-dbceb9\"/\"default-dockercfg-cnps8\""
Oct 31 19:33:34.508617 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.508596    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"]
Oct 31 19:33:34.534729 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.534694    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-tmp-dir\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.534904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.534738    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-dshm\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.534904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.534791    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b55310-85c9-403e-98de-4be997a7aeaf-tls-certs\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.534904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.534828    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-home\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.534904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.534878    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmrl\" (UniqueName: \"kubernetes.io/projected/88b55310-85c9-403e-98de-4be997a7aeaf-kube-api-access-qmmrl\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.535083 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.535059    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-model-cache\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.535190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.535169    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-tmp-dir\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.535299 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.535276    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-home\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.535514 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.535495    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-model-cache\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.537355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.537330    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-dshm\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.537702 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.537684    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b55310-85c9-403e-98de-4be997a7aeaf-tls-certs\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.592196 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.592158    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmrl\" (UniqueName: \"kubernetes.io/projected/88b55310-85c9-403e-98de-4be997a7aeaf-kube-api-access-qmmrl\") pod \"sub-test-model-dbceb9-kserve-68cd68ddff-bw85f\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") " pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:34.715199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:34.715115    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:35.095680 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.095652    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"]
Oct 31 19:33:35.097672 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:33:35.097641    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88b55310_85c9_403e_98de_4be997a7aeaf.slice/crio-067d14b6c7413ead511bfae2f825eb8f05f606f015244bd3260f3b1549d3d1e9 WatchSource:0}: Error finding container 067d14b6c7413ead511bfae2f825eb8f05f606f015244bd3260f3b1549d3d1e9: Status 404 returned error can't find the container with id 067d14b6c7413ead511bfae2f825eb8f05f606f015244bd3260f3b1549d3d1e9
Oct 31 19:33:35.559900 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.559871    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:33:35.649330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649295    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-certs\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.649580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649389    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-data\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.649580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649434    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-credential-socket\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.649580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649509    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-socket\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.649580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649547    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-podinfo\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.649827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649634    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-envoy\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.649827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649641    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-certs" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:33:35.649827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649670    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-kube-api-access-xfjg8\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-kube-api-access-xfjg8\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.649827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649726    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-data" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:33:35.649827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649748    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a3cc43-225f-4be1-96fb-490ee08eeb04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a3cc43-225f-4be1-96fb-490ee08eeb04-istiod-ca-cert\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.649827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649794    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-credential-socket" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:33:35.650153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649864    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-token\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-token\") pod \"86a3cc43-225f-4be1-96fb-490ee08eeb04\" (UID: \"86a3cc43-225f-4be1-96fb-490ee08eeb04\") "
Oct 31 19:33:35.650153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.649903    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-socket" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:33:35.650153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.650095    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a3cc43-225f-4be1-96fb-490ee08eeb04-istiod-ca-cert" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:33:35.650477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.650408    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.650477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.650427    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a3cc43-225f-4be1-96fb-490ee08eeb04-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.650477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.650437    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.650477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.650445    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.650477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.650453    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.652053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.652021    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-kube-api-access-xfjg8" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "kube-api-access-xfjg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:35.652186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.652132    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-token" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:35.652359 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.652337    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-envoy" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:33:35.652427 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.652355    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-podinfo" pod "86a3cc43-225f-4be1-96fb-490ee08eeb04" (UID: "86a3cc43-225f-4be1-96fb-490ee08eeb04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:33:35.751249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.751212    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.751249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.751243    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.751249 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.751255    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfjg8\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-kube-api-access-xfjg8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.751594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.751266    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a3cc43-225f-4be1-96fb-490ee08eeb04-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:33:35.928314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.928226    5551 generic.go:356] "Generic (PLEG): container finished" podID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerID="bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1" exitCode=0
Oct 31 19:33:35.928314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.928286    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"
Oct 31 19:33:35.928609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.928311    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" event={"ID":"86a3cc43-225f-4be1-96fb-490ee08eeb04","Type":"ContainerDied","Data":"bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1"}
Oct 31 19:33:35.928609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.928363    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv" event={"ID":"86a3cc43-225f-4be1-96fb-490ee08eeb04","Type":"ContainerDied","Data":"2b1b1ffc226a874f878b158ab3f2da44adc1043ab9d89760f9959e85a940c920"}
Oct 31 19:33:35.928609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.928391    5551 scope.go:122] "RemoveContainer" containerID="bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1"
Oct 31 19:33:35.930436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.930411    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f" event={"ID":"88b55310-85c9-403e-98de-4be997a7aeaf","Type":"ContainerStarted","Data":"38feea745597edf3a3e509f05cadadb8a5c443d068e0e220f03f754321454886"}
Oct 31 19:33:35.930530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.930469    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f" event={"ID":"88b55310-85c9-403e-98de-4be997a7aeaf","Type":"ContainerStarted","Data":"067d14b6c7413ead511bfae2f825eb8f05f606f015244bd3260f3b1549d3d1e9"}
Oct 31 19:33:35.940339 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.940323    5551 scope.go:122] "RemoveContainer" containerID="889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f"
Oct 31 19:33:35.949858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.949839    5551 scope.go:122] "RemoveContainer" containerID="bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1"
Oct 31 19:33:35.950164 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:33:35.950141    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1\": container with ID starting with bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1 not found: ID does not exist" containerID="bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1"
Oct 31 19:33:35.950259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.950171    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1"} err="failed to get container status \"bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1\": rpc error: code = NotFound desc = could not find container \"bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1\": container with ID starting with bef59b638543913edc640144d7f22990c620704f661c5b9c820df7eccb114cd1 not found: ID does not exist"
Oct 31 19:33:35.950259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.950187    5551 scope.go:122] "RemoveContainer" containerID="889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f"
Oct 31 19:33:35.950829 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:33:35.950802    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f\": container with ID starting with 889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f not found: ID does not exist" containerID="889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f"
Oct 31 19:33:35.950907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:35.950835    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f"} err="failed to get container status \"889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f\": rpc error: code = NotFound desc = could not find container \"889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f\": container with ID starting with 889c68fa0516c1017b423e2bd54cbf212f5728af2ba7821a4949c0775aa32a1f not found: ID does not exist"
Oct 31 19:33:36.093254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:36.093225    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"]
Oct 31 19:33:36.097909 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:36.097884    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fdd317-openshift-default-686994bf9-thxpv"]
Oct 31 19:33:36.208203 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:36.208101    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f" podStartSLOduration=2.208086731 podStartE2EDuration="2.208086731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:36.2067259 +0000 UTC m=+4319.437964201" watchObservedRunningTime="2026-10-31 19:33:36.208086731 +0000 UTC m=+4319.439325030"
Oct 31 19:33:36.715581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:36.715543    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:36.909777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:36.909738    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" path="/var/lib/kubelet/pods/86a3cc43-225f-4be1-96fb-490ee08eeb04/volumes"
Oct 31 19:33:38.957262 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.957232    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"]
Oct 31 19:33:38.957829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.957815    5551 cpu_manager.go:414] "containerMap: removing container" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy"
Oct 31 19:33:38.957879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.957831    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy"
Oct 31 19:33:38.957879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.957864    5551 cpu_manager.go:414] "containerMap: removing container" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy"
Oct 31 19:33:38.957879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.957868    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa859b3-2ed2-4a73-bfa6-51028ab181b1" containerName="istio-proxy"
Oct 31 19:33:38.957987 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.957953    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy"
Oct 31 19:33:38.957987 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.957968    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy"
Oct 31 19:33:38.962830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.962813    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.987763 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.987737    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-envoy\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.987893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.987796    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-podinfo\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.987957 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.987935    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-certs\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.987996 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.987974    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-token\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.988046 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.988018    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sn6h\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-kube-api-access-8sn6h\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.988181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.988158    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istiod-ca-cert\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.988531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.988451    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-socket\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.988642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.988585    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-credential-socket\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:38.988642 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:38.988636    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-data\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.026134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.023785    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-049c8941-openshift-default-dockercfg-jcvx8\""
Oct 31 19:33:39.093620 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.093571    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sn6h\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-kube-api-access-8sn6h\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.093833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.093645    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istiod-ca-cert\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.093833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.093696    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-socket\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.093833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.093728    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-credential-socket\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.093833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.093785    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-data\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.094073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.093878    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-envoy\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.094073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.093928    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-podinfo\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.094073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.094000    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-certs\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.094073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.094035    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-token\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.095303 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.095273    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istiod-ca-cert\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.095612 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.095591    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-socket\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.095854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.095831    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-credential-socket\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.096076 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.096059    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-data\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.107532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.099431    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-certs\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.109948 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.109861    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-podinfo\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.116109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.113789    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-envoy\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.151788 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.151754    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sn6h\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-kube-api-access-8sn6h\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.151961 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.151764    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-token\") pod \"e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") " pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.186481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.184698    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"]
Oct 31 19:33:39.276816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.276725    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:39.719929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.719902    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"]
Oct 31 19:33:39.720650 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:33:39.720625    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e65d3d4_2f00_47ce_999d_3c13f5d95801.slice/crio-ea7fd878bda6185b3ee2d1f02c2784f3681e0a10e072380b5da9f3b5aeac3ed8 WatchSource:0}: Error finding container ea7fd878bda6185b3ee2d1f02c2784f3681e0a10e072380b5da9f3b5aeac3ed8: Status 404 returned error can't find the container with id ea7fd878bda6185b3ee2d1f02c2784f3681e0a10e072380b5da9f3b5aeac3ed8
Oct 31 19:33:39.728535 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.728492    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:33:39.728661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.728581    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:33:39.728661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.728616    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:33:39.962283 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.962243    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" event={"ID":"1e65d3d4-2f00-47ce-999d-3c13f5d95801","Type":"ContainerStarted","Data":"382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48"}
Oct 31 19:33:39.962701 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:39.962302    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" event={"ID":"1e65d3d4-2f00-47ce-999d-3c13f5d95801","Type":"ContainerStarted","Data":"ea7fd878bda6185b3ee2d1f02c2784f3681e0a10e072380b5da9f3b5aeac3ed8"}
Oct 31 19:33:40.081600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:40.081499    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" podStartSLOduration=2.0814811029999998 podStartE2EDuration="2.081481103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:40.079837096 +0000 UTC m=+4323.311075412" watchObservedRunningTime="2026-10-31 19:33:40.081481103 +0000 UTC m=+4323.312719394"
Oct 31 19:33:40.276914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:40.276881    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:40.283306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:40.283282    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:40.970817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:40.970780    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:40.971826 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:40.971806    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:33:44.715803 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:44.715693    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:44.728267 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:44.728241    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:45.014280 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:33:45.014202    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:33:46.606679 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:33:46.606642    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:34:16.604443 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:34:16.604367    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:34:38.755704 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.755666    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"]
Oct 31 19:34:38.756234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.756220    5551 cpu_manager.go:414] "containerMap: removing container" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy"
Oct 31 19:34:38.756280 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.756235    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a3cc43-225f-4be1-96fb-490ee08eeb04" containerName="istio-proxy"
Oct 31 19:34:38.759985 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.759965    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.857029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.856993    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-model-cache\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.857198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.857153    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tmp-dir\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.857270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.857220    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5p4\" (UniqueName: \"kubernetes.io/projected/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-kube-api-access-jn5p4\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.857338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.857287    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-dshm\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.857414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.857390    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tls-certs\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.857567 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.857544    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-home\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.862132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.862111    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-d1ff32\"/\"openshift-service-ca.crt\""
Oct 31 19:34:38.862252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.862111    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-d1ff32\"/\"rate-test-model-d1ff32-kserve-self-signed-certs\""
Oct 31 19:34:38.862252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.862156    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-d1ff32\"/\"default-dockercfg-jl4rd\""
Oct 31 19:34:38.862252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.862191    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-d1ff32\"/\"kube-root-ca.crt\""
Oct 31 19:34:38.864511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.864487    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"]
Oct 31 19:34:38.958681 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.958644    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-home\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.958902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.958757    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-model-cache\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.958902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.958875    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tmp-dir\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.959034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.958927    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5p4\" (UniqueName: \"kubernetes.io/projected/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-kube-api-access-jn5p4\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.959034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.958958    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-dshm\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.959142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.959033    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tls-certs\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.959197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.959174    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-model-cache\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.959307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.959267    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tmp-dir\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.959682 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.959658    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-home\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.962159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.962135    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-dshm\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:38.962325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:38.962304    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tls-certs\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:39.020234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:39.020168    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5p4\" (UniqueName: \"kubernetes.io/projected/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-kube-api-access-jn5p4\") pod \"rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:39.072566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:39.072528    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:39.456322 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:39.456292    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"]
Oct 31 19:34:39.458016 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:34:39.457979    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ae8e0e_0b8a_44a7_a5f4_4a5b4661ea9e.slice/crio-e0800a69b901009a6c303ba1c4d213fc3f59614b29c4f70a8ef4904d1a5dea5f WatchSource:0}: Error finding container e0800a69b901009a6c303ba1c4d213fc3f59614b29c4f70a8ef4904d1a5dea5f: Status 404 returned error can't find the container with id e0800a69b901009a6c303ba1c4d213fc3f59614b29c4f70a8ef4904d1a5dea5f
Oct 31 19:34:39.525073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:39.525047    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb" event={"ID":"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e","Type":"ContainerStarted","Data":"a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516"}
Oct 31 19:34:39.525197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:39.525086    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb" event={"ID":"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e","Type":"ContainerStarted","Data":"e0800a69b901009a6c303ba1c4d213fc3f59614b29c4f70a8ef4904d1a5dea5f"}
Oct 31 19:34:39.643134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:39.643073    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb" podStartSLOduration=1.643056905 podStartE2EDuration="1.643056905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:39.641836442 +0000 UTC m=+4382.873074741" watchObservedRunningTime="2026-10-31 19:34:39.643056905 +0000 UTC m=+4382.874295193"
Oct 31 19:34:40.073721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:40.073680    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:40.506288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:40.506253    5551 scope.go:122] "RemoveContainer" containerID="5523361980ebabde5198c9f214144f010ac2621f1f7f3f1b0bd4ca5dc80a59fc"
Oct 31 19:34:46.605050 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:34:46.604984    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:34:48.745162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.745123    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"]
Oct 31 19:34:48.748931 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.748910    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.854711 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.854686    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-049c8941-serving-cert\""
Oct 31 19:34:48.877008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.876971    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"]
Oct 31 19:34:48.889233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.889193    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e4fd7b-e71b-4eff-875c-e1b3362191a8-openshift-service-ca\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.889500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.889432    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.889596 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.889503    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mws\" (UniqueName: \"kubernetes.io/projected/51e4fd7b-e71b-4eff-875c-e1b3362191a8-kube-api-access-w9mws\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.889677 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.889650    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-metrics-certs\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.994125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.994086    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e4fd7b-e71b-4eff-875c-e1b3362191a8-openshift-service-ca\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.995000 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.994970    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e4fd7b-e71b-4eff-875c-e1b3362191a8-openshift-service-ca\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.995202 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.995149    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.995202 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.995177    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mws\" (UniqueName: \"kubernetes.io/projected/51e4fd7b-e71b-4eff-875c-e1b3362191a8-kube-api-access-w9mws\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.995337 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:48.995266    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-metrics-certs\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:48.995337 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:34:48.995292    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-serving-cert: secret "maas-api-e2e-mt-049c8941-serving-cert" not found
Oct 31 19:34:48.995510 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:34:48.995366    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls podName:51e4fd7b-e71b-4eff-875c-e1b3362191a8 nodeName:}" failed. No retries permitted until 2026-10-31 19:34:49.495347959 +0000 UTC m=+4392.726586226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls") pod "maas-api-e2e-mt-049c8941-c9c75d985-ttzcr" (UID: "51e4fd7b-e71b-4eff-875c-e1b3362191a8") : secret "maas-api-e2e-mt-049c8941-serving-cert" not found
Oct 31 19:34:49.009081 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:49.009048    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-metrics-certs\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:49.058882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:49.058854    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mws\" (UniqueName: \"kubernetes.io/projected/51e4fd7b-e71b-4eff-875c-e1b3362191a8-kube-api-access-w9mws\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:49.073083 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:49.073049    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:49.086180 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:49.086154    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:49.502961 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:49.502923    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:49.505522 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:49.505502    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls\") pod \"maas-api-e2e-mt-049c8941-c9c75d985-ttzcr\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:49.611387 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:49.611359    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:34:49.667222 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:49.667177    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:50.060690 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:50.060665    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"]
Oct 31 19:34:50.061772 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:34:50.061746    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e4fd7b_e71b_4eff_875c_e1b3362191a8.slice/crio-ccbed75339256b664827e367f533f25f3cb1bc2ef998d8fa72282a95868caff9 WatchSource:0}: Error finding container ccbed75339256b664827e367f533f25f3cb1bc2ef998d8fa72282a95868caff9: Status 404 returned error can't find the container with id ccbed75339256b664827e367f533f25f3cb1bc2ef998d8fa72282a95868caff9
Oct 31 19:34:50.063255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:50.063234    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:34:50.604242 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:50.604207    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr" event={"ID":"51e4fd7b-e71b-4eff-875c-e1b3362191a8","Type":"ContainerStarted","Data":"ccbed75339256b664827e367f533f25f3cb1bc2ef998d8fa72282a95868caff9"}
Oct 31 19:34:51.613472 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:51.613425    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr" event={"ID":"51e4fd7b-e71b-4eff-875c-e1b3362191a8","Type":"ContainerStarted","Data":"9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8"}
Oct 31 19:34:51.613861 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:51.613519    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:34:51.731111 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:51.731053    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr" podStartSLOduration=2.8243550649999998 podStartE2EDuration="3.731035703s" totalImagesPullingTime="906.680638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:51.727403606 +0000 UTC m=+4394.958641896" watchObservedRunningTime="2026-10-31 19:34:51.731035703 +0000 UTC m=+4394.962273994"
Oct 31 19:34:57.627370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:34:57.627336    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:35:08.523887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.522800    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"]
Oct 31 19:35:08.529217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.528723    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.586653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.586615    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"]
Oct 31 19:35:08.611565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.611535    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6cfd68\"/\"default-dockercfg-4tflp\""
Oct 31 19:35:08.611961 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.611935    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6cfd68\"/\"kube-root-ca.crt\""
Oct 31 19:35:08.612167 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.612143    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6cfd68\"/\"openshift-service-ca.crt\""
Oct 31 19:35:08.630114 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.630094    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6cfd68\"/\"sub-test-model-6cfd68-kserve-self-signed-certs\""
Oct 31 19:35:08.709531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.708608    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-model-cache\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.709531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.708700    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8j4\" (UniqueName: \"kubernetes.io/projected/abfb49ee-7672-4e09-be02-f0dc4aa237de-kube-api-access-lw8j4\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.709531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.708868    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-dshm\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.709531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.708996    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb49ee-7672-4e09-be02-f0dc4aa237de-tls-certs\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.709531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.709032    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-home\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.709531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.709167    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-tmp-dir\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.811357 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.811251    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb49ee-7672-4e09-be02-f0dc4aa237de-tls-certs\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.811357 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.811318    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-home\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.811678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.811448    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-tmp-dir\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.811678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.811527    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-model-cache\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.811678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.811602    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8j4\" (UniqueName: \"kubernetes.io/projected/abfb49ee-7672-4e09-be02-f0dc4aa237de-kube-api-access-lw8j4\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.811855 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.811713    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-dshm\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.811924 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.811867    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-tmp-dir\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.811986 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.811968    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-model-cache\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.812141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.812118    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-home\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.814143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.814115    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-dshm\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.814959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.814939    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb49ee-7672-4e09-be02-f0dc4aa237de-tls-certs\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:08.884516 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:08.881910    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8j4\" (UniqueName: \"kubernetes.io/projected/abfb49ee-7672-4e09-be02-f0dc4aa237de-kube-api-access-lw8j4\") pod \"sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") " pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:09.142957 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:09.142919    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:09.583852 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:09.583816    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"]
Oct 31 19:35:09.805656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:09.802269    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn" event={"ID":"abfb49ee-7672-4e09-be02-f0dc4aa237de","Type":"ContainerStarted","Data":"3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b"}
Oct 31 19:35:09.805656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:09.802313    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn" event={"ID":"abfb49ee-7672-4e09-be02-f0dc4aa237de","Type":"ContainerStarted","Data":"a36fff910908389565329bf822c5370413f878bc072f8e3d8f9f2aa683d55cf9"}
Oct 31 19:35:09.920877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:09.920826    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn" podStartSLOduration=1.92081159 podStartE2EDuration="1.92081159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:09.919630901 +0000 UTC m=+4413.150869385" watchObservedRunningTime="2026-10-31 19:35:09.92081159 +0000 UTC m=+4413.152049879"
Oct 31 19:35:10.143016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:10.142978    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:15.850840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:15.850804    5551 generic.go:356] "Generic (PLEG): container finished" podID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerID="382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48" exitCode=0
Oct 31 19:35:15.851406 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:15.850895    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" event={"ID":"1e65d3d4-2f00-47ce-999d-3c13f5d95801","Type":"ContainerDied","Data":"382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48"}
Oct 31 19:35:15.851502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:15.851485    5551 scope.go:122] "RemoveContainer" containerID="382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48"
Oct 31 19:35:15.853286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:15.853259    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:35:15.853378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:15.853329    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:35:15.853378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:15.853360    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:35:16.612377 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:35:16.612343    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:35:16.888584 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:16.888543    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" event={"ID":"1e65d3d4-2f00-47ce-999d-3c13f5d95801","Type":"ContainerStarted","Data":"cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6"}
Oct 31 19:35:17.281175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:17.281035    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:35:17.281513 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:17.281486    5551 patch_prober.go:29] interesting pod/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:35:17.281717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:17.281686    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 31 19:35:18.278070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:18.278038    5551 patch_prober.go:29] interesting pod/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:35:18.278572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:18.278096    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 31 19:35:19.143201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:19.143166    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:19.183380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:19.183348    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:19.277952 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:19.277910    5551 patch_prober.go:29] interesting pod/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:35:19.278143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:19.277973    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 31 19:35:19.922717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:19.922684    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:35:20.088514 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:20.088452    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"]
Oct 31 19:35:20.091433 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:20.091404    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e4f5f90-8nhkj"]
Oct 31 19:35:20.277389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:20.277300    5551 patch_prober.go:29] interesting pod/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:35:20.277389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:20.277355    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 31 19:35:20.910004 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:20.909969    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f30872-932e-4def-89d1-979e1ab0dfae" path="/var/lib/kubelet/pods/09f30872-932e-4def-89d1-979e1ab0dfae/volumes"
Oct 31 19:35:21.277176 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:21.277069    5551 patch_prober.go:29] interesting pod/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:35:21.277176 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:21.277136    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 31 19:35:22.282527 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:22.282392    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:35:22.283026 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:22.282771    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:35:22.283429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:22.283409    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:35:46.606619 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:35:46.606581    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:35:57.864476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:57.864391    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"]
Oct 31 19:35:57.868573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:57.868544    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:57.924531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:57.924492    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"]
Oct 31 19:35:57.969257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:57.969223    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1aefcd\"/\"kube-root-ca.crt\""
Oct 31 19:35:57.969257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:57.969223    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1aefcd\"/\"openshift-service-ca.crt\""
Oct 31 19:35:57.969557 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:57.969289    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1aefcd\"/\"rate-test-model-1aefcd-kserve-self-signed-certs\""
Oct 31 19:35:57.972092 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:57.972065    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1aefcd\"/\"default-dockercfg-5c4w4\""
Oct 31 19:35:58.020379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.020334    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-tmp-dir\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.020681 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.020438    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-dshm\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.020681 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.020535    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94ebace-d451-48fa-81c6-d9970c417721-tls-certs\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.021092 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.020686    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-home\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.021092 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.020732    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzwt\" (UniqueName: \"kubernetes.io/projected/d94ebace-d451-48fa-81c6-d9970c417721-kube-api-access-8tzwt\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.021092 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.020861    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-model-cache\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.121812 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.121721    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-dshm\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.121812 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.121763    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94ebace-d451-48fa-81c6-d9970c417721-tls-certs\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.122064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.121834    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-home\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.122064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.121859    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzwt\" (UniqueName: \"kubernetes.io/projected/d94ebace-d451-48fa-81c6-d9970c417721-kube-api-access-8tzwt\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.122064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.121935    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-model-cache\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.122064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.122007    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-tmp-dir\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.122298 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.122210    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-home\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.122395 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.122372    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-model-cache\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.122492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.122406    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-tmp-dir\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.124224 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.124201    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-dshm\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.124573 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.124554    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94ebace-d451-48fa-81c6-d9970c417721-tls-certs\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.178449 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.178407    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzwt\" (UniqueName: \"kubernetes.io/projected/d94ebace-d451-48fa-81c6-d9970c417721-kube-api-access-8tzwt\") pod \"rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") " pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.183162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.183132    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:35:58.563717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:58.563689    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"]
Oct 31 19:35:58.564299 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:35:58.564273    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd94ebace_d451_48fa_81c6_d9970c417721.slice/crio-76af632035008ccdcfdd4b798b20633b9422b89362c14c12199d57c2c30b698f WatchSource:0}: Error finding container 76af632035008ccdcfdd4b798b20633b9422b89362c14c12199d57c2c30b698f: Status 404 returned error can't find the container with id 76af632035008ccdcfdd4b798b20633b9422b89362c14c12199d57c2c30b698f
Oct 31 19:35:59.201320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:59.201282    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p" event={"ID":"d94ebace-d451-48fa-81c6-d9970c417721","Type":"ContainerStarted","Data":"00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36"}
Oct 31 19:35:59.201765 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:59.201330    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p" event={"ID":"d94ebace-d451-48fa-81c6-d9970c417721","Type":"ContainerStarted","Data":"76af632035008ccdcfdd4b798b20633b9422b89362c14c12199d57c2c30b698f"}
Oct 31 19:35:59.314582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:35:59.314513    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p" podStartSLOduration=2.3144932 podStartE2EDuration="2.3144932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:59.314429694 +0000 UTC m=+4462.545667988" watchObservedRunningTime="2026-10-31 19:35:59.3144932 +0000 UTC m=+4462.545731491"
Oct 31 19:36:00.183502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:36:00.183428    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:36:08.184119 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:36:08.184078    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:36:08.197658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:36:08.197633    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:36:08.275062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:36:08.275033    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:36:16.607862 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:36:16.607826    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:36:40.552118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:36:40.552082    5551 scope.go:122] "RemoveContainer" containerID="a0ed80fc7c2dabb0bcb899f6530a62a4bbb7d710ad8d78069abc130d80b3f730"
Oct 31 19:36:40.613530 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:36:40.613507    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:36:40.613971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:36:40.613949    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:36:46.607696 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:36:46.607618    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:37:16.608614 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:37:16.608564    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:37:40.113316 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.113275    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"]
Oct 31 19:37:40.113949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.113540    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv" podUID="c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" containerName="maas-api" containerID="cri-o://cb0cab60c64867baf3cf8dfa3be97537849d9c8c567d695fed0fb230ad692323" gracePeriod=30
Oct 31 19:37:40.413980 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.413952    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:37:40.483054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.481498    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-openshift-service-ca\") pod \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") "
Oct 31 19:37:40.483054 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.481578    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-kube-api-access-h2j9t\" (UniqueName: \"kubernetes.io/projected/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-kube-api-access-h2j9t\") pod \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") "
Oct 31 19:37:40.484377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.483525    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls\") pod \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") "
Oct 31 19:37:40.484377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.483725    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-metrics-certs\") pod \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\" (UID: \"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8\") "
Oct 31 19:37:40.487187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.487156    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-openshift-service-ca" pod "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" (UID: "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:37:40.494730 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.494645    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls" pod "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" (UID: "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:37:40.497271 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.495805    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-kube-api-access-h2j9t" pod "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" (UID: "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8"). InnerVolumeSpecName "kube-api-access-h2j9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:40.526479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.523663    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-metrics-certs" pod "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" (UID: "c962f3f6-c8e1-4e5b-8e77-512a5e7879f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:37:40.586598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.586562    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:40.586598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.586592    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:40.586598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.586602    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2j9t\" (UniqueName: \"kubernetes.io/projected/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-kube-api-access-h2j9t\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:40.586843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.586612    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:40.586843 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:40.586719    5551 scope.go:122] "RemoveContainer" containerID="cb0cab60c64867baf3cf8dfa3be97537849d9c8c567d695fed0fb230ad692323"
Oct 31 19:37:41.027118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:41.027072    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv" event={"ID":"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8","Type":"ContainerDied","Data":"cb0cab60c64867baf3cf8dfa3be97537849d9c8c567d695fed0fb230ad692323"}
Oct 31 19:37:41.027331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:41.027146    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv" event={"ID":"c962f3f6-c8e1-4e5b-8e77-512a5e7879f8","Type":"ContainerDied","Data":"9f7ea9c3166eef8591310db094c438c48b6ac612028bf4c1519c00b30e46945c"}
Oct 31 19:37:41.027331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:41.027087    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"
Oct 31 19:37:41.266203 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:41.266098    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"]
Oct 31 19:37:41.269959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:41.269933    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d8f34e-5bd77d8645-649qv"]
Oct 31 19:37:42.909446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:42.909413    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" path="/var/lib/kubelet/pods/c962f3f6-c8e1-4e5b-8e77-512a5e7879f8/volumes"
Oct 31 19:37:46.521058 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.521025    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"]
Oct 31 19:37:46.521676 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.521658    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" containerName="maas-api"
Oct 31 19:37:46.521728 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.521678    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" containerName="maas-api"
Oct 31 19:37:46.521817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.521807    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c962f3f6-c8e1-4e5b-8e77-512a5e7879f8" containerName="maas-api"
Oct 31 19:37:46.525305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.525288    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.565949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.565914    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqp7c\" (UniqueName: \"kubernetes.io/projected/ee395a46-1e79-45e8-948f-b714877ff218-kube-api-access-cqp7c\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.566099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.566076    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee395a46-1e79-45e8-948f-b714877ff218-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.566150 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.566135    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.566259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.566233    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.577287 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.577264    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-0f794f-serving-cert\""
Oct 31 19:37:46.612700 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:37:46.612662    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:37:46.632281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.632243    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"]
Oct 31 19:37:46.667817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.667789    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqp7c\" (UniqueName: \"kubernetes.io/projected/ee395a46-1e79-45e8-948f-b714877ff218-kube-api-access-cqp7c\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.668002 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.667952    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee395a46-1e79-45e8-948f-b714877ff218-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.668072 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.668039    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.668134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.668070    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.668532 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:37:46.668506    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-serving-cert: secret "maas-api-e2e-shared-a-w1-0f794f-serving-cert" not found
Oct 31 19:37:46.668665 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:37:46.668581    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls podName:ee395a46-1e79-45e8-948f-b714877ff218 nodeName:}" failed. No retries permitted until 2026-10-31 19:37:47.16855809 +0000 UTC m=+4570.399796366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls") pod "maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx" (UID: "ee395a46-1e79-45e8-948f-b714877ff218") : secret "maas-api-e2e-shared-a-w1-0f794f-serving-cert" not found
Oct 31 19:37:46.669030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.668929    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee395a46-1e79-45e8-948f-b714877ff218-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.670967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.670938    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:46.728055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:46.728024    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqp7c\" (UniqueName: \"kubernetes.io/projected/ee395a46-1e79-45e8-948f-b714877ff218-kube-api-access-cqp7c\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:47.177243 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:47.177204    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:47.180192 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:47.180159    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:47.440653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:47.440556    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:47.826864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:47.826838    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"]
Oct 31 19:37:47.827570 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:37:47.827541    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee395a46_1e79_45e8_948f_b714877ff218.slice/crio-75b8514bffd4adb90e4429ea384c8c8ce93405d82fcbeee71f4d85d8589b9a99 WatchSource:0}: Error finding container 75b8514bffd4adb90e4429ea384c8c8ce93405d82fcbeee71f4d85d8589b9a99: Status 404 returned error can't find the container with id 75b8514bffd4adb90e4429ea384c8c8ce93405d82fcbeee71f4d85d8589b9a99
Oct 31 19:37:48.083075 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:48.082984    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx" event={"ID":"ee395a46-1e79-45e8-948f-b714877ff218","Type":"ContainerStarted","Data":"75b8514bffd4adb90e4429ea384c8c8ce93405d82fcbeee71f4d85d8589b9a99"}
Oct 31 19:37:49.092289 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:49.092196    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx" event={"ID":"ee395a46-1e79-45e8-948f-b714877ff218","Type":"ContainerStarted","Data":"5192d7f1694b548cb52631eb5da41fcf558760e1169a4c4ccd61fb6dcbdece14"}
Oct 31 19:37:49.092747 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:49.092307    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:49.221663 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:49.221603    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx" podStartSLOduration=2.3338357800000002 podStartE2EDuration="3.221587976s" totalImagesPullingTime="887.752196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:49.219890564 +0000 UTC m=+4572.451128855" watchObservedRunningTime="2026-10-31 19:37:49.221587976 +0000 UTC m=+4572.452826280"
Oct 31 19:37:50.427268 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:50.427228    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"]
Oct 31 19:37:50.427784 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:50.427470    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy" containerID="cri-o://f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118" gracePeriod=30
Oct 31 19:37:55.105155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.105127    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:37:55.712034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.711973    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:37:55.815816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.815782    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-data\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.815816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.815817    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77c2c425-0d2f-4073-a253-808f42309478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c2c425-0d2f-4073-a253-808f42309478-istio-podinfo\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.816090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.815874    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c2c425-0d2f-4073-a253-808f42309478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c2c425-0d2f-4073-a253-808f42309478-istiod-ca-cert\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.816090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.815903    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-envoy\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.816090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.815960    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-istio-token\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-istio-token\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.816090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816040    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-socket\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.816323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816094    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-certs\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.816323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816145    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-data" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:37:55.816323 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816294    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-kube-api-access-62tpg\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-kube-api-access-62tpg\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.816640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816359    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-credential-socket\") pod \"77c2c425-0d2f-4073-a253-808f42309478\" (UID: \"77c2c425-0d2f-4073-a253-808f42309478\") "
Oct 31 19:37:55.816640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816380    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c2c425-0d2f-4073-a253-808f42309478-istiod-ca-cert" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:37:55.816640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816538    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-socket" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:37:55.816640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816595    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-certs" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:37:55.816947 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.816736    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-credential-socket" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:37:55.817211 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.817180    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:55.817211 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.817207    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:55.817368 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.817222    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c2c425-0d2f-4073-a253-808f42309478-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:55.817368 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.817238    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:55.817368 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.817252    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:55.818750 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.818720    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-istio-token" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:55.818898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.818747    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77c2c425-0d2f-4073-a253-808f42309478-istio-podinfo" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:37:55.819162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.819133    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-kube-api-access-62tpg" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "kube-api-access-62tpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:55.819162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.819150    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-envoy" pod "77c2c425-0d2f-4073-a253-808f42309478" (UID: "77c2c425-0d2f-4073-a253-808f42309478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:37:55.919277 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.919239    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62tpg\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-kube-api-access-62tpg\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:55.919499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.919281    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c2c425-0d2f-4073-a253-808f42309478-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:55.919499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.919302    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c2c425-0d2f-4073-a253-808f42309478-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:55.919499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:55.919315    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c2c425-0d2f-4073-a253-808f42309478-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:56.143905 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.143872    5551 generic.go:356] "Generic (PLEG): container finished" podID="77c2c425-0d2f-4073-a253-808f42309478" containerID="f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118" exitCode=0
Oct 31 19:37:56.144326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.143929    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"
Oct 31 19:37:56.144326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.143964    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" event={"ID":"77c2c425-0d2f-4073-a253-808f42309478","Type":"ContainerDied","Data":"f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118"}
Oct 31 19:37:56.144326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.144023    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n" event={"ID":"77c2c425-0d2f-4073-a253-808f42309478","Type":"ContainerDied","Data":"23bf3956859c0b8afb4f3b430b2f626498ebb9b8e27eeacfb23ea2b80adc2f28"}
Oct 31 19:37:56.144326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.144056    5551 scope.go:122] "RemoveContainer" containerID="f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118"
Oct 31 19:37:56.156574 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.156547    5551 scope.go:122] "RemoveContainer" containerID="7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766"
Oct 31 19:37:56.168824 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.168777    5551 scope.go:122] "RemoveContainer" containerID="f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118"
Oct 31 19:37:56.175122 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:37:56.174941    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118\": container with ID starting with f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118 not found: ID does not exist" containerID="f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118"
Oct 31 19:37:56.175122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.174994    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118"} err="failed to get container status \"f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118\": rpc error: code = NotFound desc = could not find container \"f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118\": container with ID starting with f4d6f1fb655a7e6a6bf57d162ec63062ef6cd04616dfacd3eea54e0c69b68118 not found: ID does not exist"
Oct 31 19:37:56.175122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.175023    5551 scope.go:122] "RemoveContainer" containerID="7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766"
Oct 31 19:37:56.175448 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:37:56.175383    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766\": container with ID starting with 7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766 not found: ID does not exist" containerID="7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766"
Oct 31 19:37:56.175448 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.175411    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766"} err="failed to get container status \"7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766\": rpc error: code = NotFound desc = could not find container \"7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766\": container with ID starting with 7f00d347ddff0d48bb4b56d5e2ce05a9c265f3ba68c80f872eb791266f720766 not found: ID does not exist"
Oct 31 19:37:56.309996 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.309928    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"]
Oct 31 19:37:56.315785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.315757    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d8f34e-openshift-default-857cb5cd5f-kcq5n"]
Oct 31 19:37:56.917336 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:37:56.917303    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c2c425-0d2f-4073-a253-808f42309478" path="/var/lib/kubelet/pods/77c2c425-0d2f-4073-a253-808f42309478/volumes"
Oct 31 19:38:16.642859 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:38:16.642776    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:38:25.619414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:25.619374    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"]
Oct 31 19:38:25.619916 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:25.619630    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr" podUID="51e4fd7b-e71b-4eff-875c-e1b3362191a8" containerName="maas-api" containerID="cri-o://9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8" gracePeriod=30
Oct 31 19:38:25.915782 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:25.913888    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:38:26.017644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.017334    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-metrics-certs\") pod \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") "
Oct 31 19:38:26.017644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.017397    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e4fd7b-e71b-4eff-875c-e1b3362191a8-kube-api-access-w9mws\" (UniqueName: \"kubernetes.io/projected/51e4fd7b-e71b-4eff-875c-e1b3362191a8-kube-api-access-w9mws\") pod \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") "
Oct 31 19:38:26.017644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.017439    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls\") pod \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") "
Oct 31 19:38:26.018515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.017791    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e4fd7b-e71b-4eff-875c-e1b3362191a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e4fd7b-e71b-4eff-875c-e1b3362191a8-openshift-service-ca\") pod \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\" (UID: \"51e4fd7b-e71b-4eff-875c-e1b3362191a8\") "
Oct 31 19:38:26.020693 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.020646    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e4fd7b-e71b-4eff-875c-e1b3362191a8-openshift-service-ca" pod "51e4fd7b-e71b-4eff-875c-e1b3362191a8" (UID: "51e4fd7b-e71b-4eff-875c-e1b3362191a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:38:26.027849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.027807    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls" pod "51e4fd7b-e71b-4eff-875c-e1b3362191a8" (UID: "51e4fd7b-e71b-4eff-875c-e1b3362191a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:38:26.061483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.041449    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-metrics-certs" pod "51e4fd7b-e71b-4eff-875c-e1b3362191a8" (UID: "51e4fd7b-e71b-4eff-875c-e1b3362191a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:38:26.093949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.078139    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e4fd7b-e71b-4eff-875c-e1b3362191a8-kube-api-access-w9mws" pod "51e4fd7b-e71b-4eff-875c-e1b3362191a8" (UID: "51e4fd7b-e71b-4eff-875c-e1b3362191a8"). InnerVolumeSpecName "kube-api-access-w9mws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:38:26.121174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.121074    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:26.121174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.121115    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9mws\" (UniqueName: \"kubernetes.io/projected/51e4fd7b-e71b-4eff-875c-e1b3362191a8-kube-api-access-w9mws\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:26.121174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.121132    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e4fd7b-e71b-4eff-875c-e1b3362191a8-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:26.121174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.121146    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e4fd7b-e71b-4eff-875c-e1b3362191a8-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:26.409027 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.408988    5551 generic.go:356] "Generic (PLEG): container finished" podID="51e4fd7b-e71b-4eff-875c-e1b3362191a8" containerID="9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8" exitCode=0
Oct 31 19:38:26.409220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.409057    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"
Oct 31 19:38:26.409220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.409073    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr" event={"ID":"51e4fd7b-e71b-4eff-875c-e1b3362191a8","Type":"ContainerDied","Data":"9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8"}
Oct 31 19:38:26.409220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.409184    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr" event={"ID":"51e4fd7b-e71b-4eff-875c-e1b3362191a8","Type":"ContainerDied","Data":"ccbed75339256b664827e367f533f25f3cb1bc2ef998d8fa72282a95868caff9"}
Oct 31 19:38:26.409220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.409214    5551 scope.go:122] "RemoveContainer" containerID="9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8"
Oct 31 19:38:26.421403 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.421366    5551 scope.go:122] "RemoveContainer" containerID="9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8"
Oct 31 19:38:26.421732 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:38:26.421711    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8\": container with ID starting with 9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8 not found: ID does not exist" containerID="9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8"
Oct 31 19:38:26.421802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.421744    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8"} err="failed to get container status \"9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8\": rpc error: code = NotFound desc = could not find container \"9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8\": container with ID starting with 9c8f64299db980ccabe1fc33ca8822a5b47add18227c7d87bfda9414e71491f8 not found: ID does not exist"
Oct 31 19:38:26.578953 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.578917    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"]
Oct 31 19:38:26.590246 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.590124    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-049c8941-c9c75d985-ttzcr"]
Oct 31 19:38:26.910803 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:26.910770    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e4fd7b-e71b-4eff-875c-e1b3362191a8" path="/var/lib/kubelet/pods/51e4fd7b-e71b-4eff-875c-e1b3362191a8/volumes"
Oct 31 19:38:38.717484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:38.717047    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"]
Oct 31 19:38:38.717484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:38.717308    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy" containerID="cri-o://cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6" gracePeriod=30
Oct 31 19:38:44.035291 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.035251    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:38:44.194151 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194123    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-credential-socket\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194184    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-envoy\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194228    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-token\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-token\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194265    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-data\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194344 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194292    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-podinfo\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194429    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-socket\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194434    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-credential-socket" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:38:44.194621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194516    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-kube-api-access-8sn6h\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-kube-api-access-8sn6h\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194577    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-certs\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194606    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istiod-ca-cert\") pod \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\" (UID: \"1e65d3d4-2f00-47ce-999d-3c13f5d95801\") "
Oct 31 19:38:44.194936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194672    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-data" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:38:44.194936 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.194849    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-socket" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:38:44.195181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.195156    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-certs" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:38:44.195267 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.195165    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istiod-ca-cert" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:38:44.195315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.195301    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.195360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.195319    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.195360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.195335    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.195360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.195350    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.197410 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.197367    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-kube-api-access-8sn6h" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "kube-api-access-8sn6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:38:44.197755 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.197730    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-envoy" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:38:44.197933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.197904    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-podinfo" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:38:44.197933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.197916    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-token" pod "1e65d3d4-2f00-47ce-999d-3c13f5d95801" (UID: "1e65d3d4-2f00-47ce-999d-3c13f5d95801"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:38:44.297109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.297009    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sn6h\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-kube-api-access-8sn6h\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.297109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.297051    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.297109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.297072    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.297109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.297087    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.297109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.297113    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e65d3d4-2f00-47ce-999d-3c13f5d95801-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:38:44.562597 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.562508    5551 generic.go:356] "Generic (PLEG): container finished" podID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerID="cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6" exitCode=0
Oct 31 19:38:44.562764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.562625    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"
Oct 31 19:38:44.562764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.562654    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" event={"ID":"1e65d3d4-2f00-47ce-999d-3c13f5d95801","Type":"ContainerDied","Data":"cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6"}
Oct 31 19:38:44.562764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.562717    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6" event={"ID":"1e65d3d4-2f00-47ce-999d-3c13f5d95801","Type":"ContainerDied","Data":"ea7fd878bda6185b3ee2d1f02c2784f3681e0a10e072380b5da9f3b5aeac3ed8"}
Oct 31 19:38:44.562764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.562740    5551 scope.go:122] "RemoveContainer" containerID="cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6"
Oct 31 19:38:44.575956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.575939    5551 scope.go:122] "RemoveContainer" containerID="382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48"
Oct 31 19:38:44.587520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.587501    5551 scope.go:122] "RemoveContainer" containerID="cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6"
Oct 31 19:38:44.587848 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:38:44.587825    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6\": container with ID starting with cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6 not found: ID does not exist" containerID="cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6"
Oct 31 19:38:44.587970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.587854    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6"} err="failed to get container status \"cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6\": rpc error: code = NotFound desc = could not find container \"cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6\": container with ID starting with cb7519d7ec5593c11291547b56b8694c5a721e71a70c5bab9a18af22d4ae94f6 not found: ID does not exist"
Oct 31 19:38:44.587970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.587873    5551 scope.go:122] "RemoveContainer" containerID="382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48"
Oct 31 19:38:44.588157 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:38:44.588140    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48\": container with ID starting with 382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48 not found: ID does not exist" containerID="382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48"
Oct 31 19:38:44.588220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.588165    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48"} err="failed to get container status \"382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48\": rpc error: code = NotFound desc = could not find container \"382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48\": container with ID starting with 382dddfa74fc19559c7c52e897d1a5cb94a53f2696479b86796aa9cebe7c7b48 not found: ID does not exist"
Oct 31 19:38:44.726973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.726941    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"]
Oct 31 19:38:44.776526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.776487    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-049c8941-openshift-default-7fbbdbf7b9-v7pq6"]
Oct 31 19:38:44.909445 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:38:44.909413    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" path="/var/lib/kubelet/pods/1e65d3d4-2f00-47ce-999d-3c13f5d95801/volumes"
Oct 31 19:38:46.633131 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:38:46.632669    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:39:06.333704 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.333668    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"]
Oct 31 19:39:06.334373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334353    5551 cpu_manager.go:414] "containerMap: removing container" podUID="51e4fd7b-e71b-4eff-875c-e1b3362191a8" containerName="maas-api"
Oct 31 19:39:06.334426 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334375    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e4fd7b-e71b-4eff-875c-e1b3362191a8" containerName="maas-api"
Oct 31 19:39:06.334520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334429    5551 cpu_manager.go:414] "containerMap: removing container" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy"
Oct 31 19:39:06.334520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334435    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy"
Oct 31 19:39:06.334520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334448    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy"
Oct 31 19:39:06.334520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334469    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy"
Oct 31 19:39:06.334520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334478    5551 cpu_manager.go:414] "containerMap: removing container" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy"
Oct 31 19:39:06.334520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334486    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy"
Oct 31 19:39:06.334520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334502    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy"
Oct 31 19:39:06.334520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334507    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy"
Oct 31 19:39:06.334811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334613    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy"
Oct 31 19:39:06.334811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334628    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy"
Oct 31 19:39:06.334811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334638    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c2c425-0d2f-4073-a253-808f42309478" containerName="istio-proxy"
Oct 31 19:39:06.334811 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.334650    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e4fd7b-e71b-4eff-875c-e1b3362191a8" containerName="maas-api"
Oct 31 19:39:06.339740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.339713    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.409477 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.409434    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-0f0e4e-openshift-default-dockercfg-vh4wk\""
Oct 31 19:39:06.446743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.446711    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"]
Oct 31 19:39:06.534703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.534656    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-certs\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.534951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.534759    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-data\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.534951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.534845    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-socket\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.534951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.534897    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-credential-socket\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.534951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.534923    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmz7c\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-kube-api-access-mmz7c\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.535198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.534962    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611474c7-427f-438e-83ed-257b09db4ff1-istio-podinfo\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.535198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.535086    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-istio-token\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.535198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.535113    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-envoy\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.535198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.535159    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611474c7-427f-438e-83ed-257b09db4ff1-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636018    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-socket\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636067    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-credential-socket\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636084    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmz7c\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-kube-api-access-mmz7c\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636108    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611474c7-427f-438e-83ed-257b09db4ff1-istio-podinfo\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636184    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-istio-token\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636203    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-envoy\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636231    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611474c7-427f-438e-83ed-257b09db4ff1-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636289    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-certs\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636338    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-data\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636713    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-data\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.636907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.636892    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-socket\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.637070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.637057    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-credential-socket\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.637963 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.637929    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-certs\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.638174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.638150    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611474c7-427f-438e-83ed-257b09db4ff1-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.640379 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.640346    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611474c7-427f-438e-83ed-257b09db4ff1-istio-podinfo\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.640681 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.640660    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-envoy\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.708382 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.708339    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-istio-token\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.708594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.708451    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmz7c\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-kube-api-access-mmz7c\") pod \"e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") " pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:06.955888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:06.955803    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:07.475399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:07.475366    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"]
Oct 31 19:39:07.476519 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:39:07.476487    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611474c7_427f_438e_83ed_257b09db4ff1.slice/crio-ab68f911df1df9469fea57ece9d41dd36e2262a21e2ad8884e8c6e248b90a314 WatchSource:0}: Error finding container ab68f911df1df9469fea57ece9d41dd36e2262a21e2ad8884e8c6e248b90a314: Status 404 returned error can't find the container with id ab68f911df1df9469fea57ece9d41dd36e2262a21e2ad8884e8c6e248b90a314
Oct 31 19:39:07.479592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:07.479551    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:39:07.479717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:07.479643    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:39:07.479717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:07.479685    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:39:07.737737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:07.737641    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" event={"ID":"611474c7-427f-438e-83ed-257b09db4ff1","Type":"ContainerStarted","Data":"1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97"}
Oct 31 19:39:07.737737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:07.737701    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" event={"ID":"611474c7-427f-438e-83ed-257b09db4ff1","Type":"ContainerStarted","Data":"ab68f911df1df9469fea57ece9d41dd36e2262a21e2ad8884e8c6e248b90a314"}
Oct 31 19:39:07.864070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:07.862636    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" podStartSLOduration=1.8626150460000002 podStartE2EDuration="1.862615046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:07.856209982 +0000 UTC m=+4651.087448271" watchObservedRunningTime="2026-10-31 19:39:07.862615046 +0000 UTC m=+4651.093853338"
Oct 31 19:39:07.955921 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:07.955885    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:08.963517 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:08.963483    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:09.749979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:09.749942    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:09.751045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:09.751019    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:39:14.224682 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.224642    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"]
Oct 31 19:39:14.225182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.224898    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp" podUID="a777b590-d839-4db9-a4c4-8cb87403f3b7" containerName="maas-api" containerID="cri-o://6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b" gracePeriod=30
Oct 31 19:39:14.498315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.498286    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:39:14.651619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.651589    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a777b590-d839-4db9-a4c4-8cb87403f3b7-kube-api-access-wvqwr\" (UniqueName: \"kubernetes.io/projected/a777b590-d839-4db9-a4c4-8cb87403f3b7-kube-api-access-wvqwr\") pod \"a777b590-d839-4db9-a4c4-8cb87403f3b7\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") "
Oct 31 19:39:14.651829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.651661    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls\") pod \"a777b590-d839-4db9-a4c4-8cb87403f3b7\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") "
Oct 31 19:39:14.651904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.651873    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a777b590-d839-4db9-a4c4-8cb87403f3b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777b590-d839-4db9-a4c4-8cb87403f3b7-openshift-service-ca\") pod \"a777b590-d839-4db9-a4c4-8cb87403f3b7\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") "
Oct 31 19:39:14.651975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.651960    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-metrics-certs\") pod \"a777b590-d839-4db9-a4c4-8cb87403f3b7\" (UID: \"a777b590-d839-4db9-a4c4-8cb87403f3b7\") "
Oct 31 19:39:14.652632 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.652298    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a777b590-d839-4db9-a4c4-8cb87403f3b7-openshift-service-ca" pod "a777b590-d839-4db9-a4c4-8cb87403f3b7" (UID: "a777b590-d839-4db9-a4c4-8cb87403f3b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:14.652632 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.652601    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a777b590-d839-4db9-a4c4-8cb87403f3b7-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:14.654139 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.654110    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a777b590-d839-4db9-a4c4-8cb87403f3b7-kube-api-access-wvqwr" pod "a777b590-d839-4db9-a4c4-8cb87403f3b7" (UID: "a777b590-d839-4db9-a4c4-8cb87403f3b7"). InnerVolumeSpecName "kube-api-access-wvqwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:14.654635 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.654613    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-metrics-certs" pod "a777b590-d839-4db9-a4c4-8cb87403f3b7" (UID: "a777b590-d839-4db9-a4c4-8cb87403f3b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:39:14.654728 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.654700    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls" pod "a777b590-d839-4db9-a4c4-8cb87403f3b7" (UID: "a777b590-d839-4db9-a4c4-8cb87403f3b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:39:14.755272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.755133    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:14.755272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.755173    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvqwr\" (UniqueName: \"kubernetes.io/projected/a777b590-d839-4db9-a4c4-8cb87403f3b7-kube-api-access-wvqwr\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:14.755272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.755190    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a777b590-d839-4db9-a4c4-8cb87403f3b7-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:14.835427 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.835390    5551 generic.go:356] "Generic (PLEG): container finished" podID="a777b590-d839-4db9-a4c4-8cb87403f3b7" containerID="6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b" exitCode=0
Oct 31 19:39:14.835645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.835438    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"
Oct 31 19:39:14.835645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.835500    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp" event={"ID":"a777b590-d839-4db9-a4c4-8cb87403f3b7","Type":"ContainerDied","Data":"6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b"}
Oct 31 19:39:14.835645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.835559    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp" event={"ID":"a777b590-d839-4db9-a4c4-8cb87403f3b7","Type":"ContainerDied","Data":"cebb1b2e9aaffee97a4104bc5c42280eacd6721818e0eba9bb2ff1c876915624"}
Oct 31 19:39:14.835645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.835590    5551 scope.go:122] "RemoveContainer" containerID="6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b"
Oct 31 19:39:14.847384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.847366    5551 scope.go:122] "RemoveContainer" containerID="6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b"
Oct 31 19:39:14.847738 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:39:14.847715    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b\": container with ID starting with 6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b not found: ID does not exist" containerID="6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b"
Oct 31 19:39:14.847806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:14.847749    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b"} err="failed to get container status \"6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b\": rpc error: code = NotFound desc = could not find container \"6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b\": container with ID starting with 6f0a84037a1b164a3839ea291b2e269035a78c7362341786de2b12f9ec935c2b not found: ID does not exist"
Oct 31 19:39:15.039539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:15.036704    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"]
Oct 31 19:39:15.062637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:15.041283    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bbd8c6-77dc997d4c-h8tfp"]
Oct 31 19:39:16.607969 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:39:16.607929    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:39:16.911432 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:16.911394    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a777b590-d839-4db9-a4c4-8cb87403f3b7" path="/var/lib/kubelet/pods/a777b590-d839-4db9-a4c4-8cb87403f3b7/volumes"
Oct 31 19:39:17.135690 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:17.135655    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"]
Oct 31 19:39:17.135897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:17.135870    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy" containerID="cri-o://f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5" gracePeriod=30
Oct 31 19:39:18.150934 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:18.150900    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"]
Oct 31 19:39:19.996851 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:19.996816    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"]
Oct 31 19:39:19.997372 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:19.997067    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy" containerID="cri-o://d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b" gracePeriod=30
Oct 31 19:39:22.395432 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.395406    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:39:22.535774 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.535694    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-podinfo\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.535774 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.535767    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d227addd-3b02-4f7d-aed0-a681fd8d680d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d227addd-3b02-4f7d-aed0-a681fd8d680d-istiod-ca-cert\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.536014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.535790    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-socket\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.536014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.535821    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-envoy\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.536014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.535915    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-data\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.536014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.535952    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-certs\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.536014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.535991    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-credential-socket\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.536290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536093    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-token\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-token\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.536290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536128    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-socket" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:22.536290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536158    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-kube-api-access-2plxj\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-kube-api-access-2plxj\") pod \"d227addd-3b02-4f7d-aed0-a681fd8d680d\" (UID: \"d227addd-3b02-4f7d-aed0-a681fd8d680d\") "
Oct 31 19:39:22.536290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536261    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-credential-socket" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:22.536562 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536298    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d227addd-3b02-4f7d-aed0-a681fd8d680d-istiod-ca-cert" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:22.536562 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536400    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-certs" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:22.536562 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536430    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-data" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:22.536739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536719    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d227addd-3b02-4f7d-aed0-a681fd8d680d-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.536739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536732    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.537010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536743    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.537010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536755    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.537010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.536769    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.538208 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.538180    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-podinfo" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:22.538338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.538313    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-envoy" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:22.538502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.538449    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-token" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:22.538694 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.538673    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-kube-api-access-2plxj" pod "d227addd-3b02-4f7d-aed0-a681fd8d680d" (UID: "d227addd-3b02-4f7d-aed0-a681fd8d680d"). InnerVolumeSpecName "kube-api-access-2plxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:22.637974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.637926    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.637974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.637966    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2plxj\" (UniqueName: \"kubernetes.io/projected/d227addd-3b02-4f7d-aed0-a681fd8d680d-kube-api-access-2plxj\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.638233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.637982    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.638233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.637997    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d227addd-3b02-4f7d-aed0-a681fd8d680d-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.903333 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.903298    5551 generic.go:356] "Generic (PLEG): container finished" podID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerID="f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5" exitCode=0
Oct 31 19:39:22.903514 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.903417    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"
Oct 31 19:39:22.917112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.917069    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" event={"ID":"d227addd-3b02-4f7d-aed0-a681fd8d680d","Type":"ContainerDied","Data":"f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5"}
Oct 31 19:39:22.917316 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.917137    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2" event={"ID":"d227addd-3b02-4f7d-aed0-a681fd8d680d","Type":"ContainerDied","Data":"a671ac2f6d8496abe3d64d6d18d7551aed56b2771fa81d9c1ba85df39c2dd4be"}
Oct 31 19:39:22.917316 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.917173    5551 scope.go:122] "RemoveContainer" containerID="f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5"
Oct 31 19:39:22.932542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.932521    5551 scope.go:122] "RemoveContainer" containerID="f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5"
Oct 31 19:39:22.932823 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:39:22.932800    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5\": container with ID starting with f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5 not found: ID does not exist" containerID="f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5"
Oct 31 19:39:22.932891 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:22.932830    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5"} err="failed to get container status \"f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5\": rpc error: code = NotFound desc = could not find container \"f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5\": container with ID starting with f2419b3c4ca75d07f43be2e9ded10053b3d203e378f6dacb8d1947ddc9c4d3c5 not found: ID does not exist"
Oct 31 19:39:23.086027 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:23.085994    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"]
Oct 31 19:39:23.090554 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:23.090524    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9ckv2"]
Oct 31 19:39:24.910802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:24.910763    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" path="/var/lib/kubelet/pods/d227addd-3b02-4f7d-aed0-a681fd8d680d/volumes"
Oct 31 19:39:25.263474 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.263433    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:39:25.370716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.370686    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-credential-socket\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.370913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.370729    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istiod-ca-cert\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.370913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.370759    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-kube-api-access-g4qm2\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.370913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.370817    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-token\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-token\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.370913 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.370896    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-envoy\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.371153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.370978    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-socket\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.371153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371038    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-podinfo\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.371153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371038    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-credential-socket" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:25.371153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371084    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-data\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.371153 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371115    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-certs\") pod \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\" (UID: \"c3ff86d6-0892-42f9-b0b9-d3319a207fcb\") "
Oct 31 19:39:25.371414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371271    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-socket" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:25.371414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371292    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istiod-ca-cert" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:25.371528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371421    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-data" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:25.371528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371475    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-certs" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:25.371802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.371779    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.372144 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.372118    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.372261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.372148    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.372261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.372164    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.372261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.372178    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.373290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.373269    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-envoy" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:25.373380 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.373357    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-kube-api-access-g4qm2" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "kube-api-access-g4qm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:25.373473 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.373436    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-podinfo" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:25.373556 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.373533    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-token" pod "c3ff86d6-0892-42f9-b0b9-d3319a207fcb" (UID: "c3ff86d6-0892-42f9-b0b9-d3319a207fcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:25.473850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.473771    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.473850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.473797    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.473850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.473811    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-kube-api-access-g4qm2\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.473850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.473827    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ff86d6-0892-42f9-b0b9-d3319a207fcb-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:25.930728 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.930689    5551 generic.go:356] "Generic (PLEG): container finished" podID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerID="d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b" exitCode=0
Oct 31 19:39:25.931247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.930779    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" event={"ID":"c3ff86d6-0892-42f9-b0b9-d3319a207fcb","Type":"ContainerDied","Data":"d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b"}
Oct 31 19:39:25.931247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.930854    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj" event={"ID":"c3ff86d6-0892-42f9-b0b9-d3319a207fcb","Type":"ContainerDied","Data":"62605799e230e3b9ce06b6415807b9f5c73fa4465da176e9236c682c1c76bd3f"}
Oct 31 19:39:25.931247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.930889    5551 scope.go:122] "RemoveContainer" containerID="d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b"
Oct 31 19:39:25.931247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.930789    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"
Oct 31 19:39:25.942328 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.942308    5551 scope.go:122] "RemoveContainer" containerID="94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315"
Oct 31 19:39:25.951807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.951788    5551 scope.go:122] "RemoveContainer" containerID="d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b"
Oct 31 19:39:25.952072 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:39:25.952053    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b\": container with ID starting with d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b not found: ID does not exist" containerID="d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b"
Oct 31 19:39:25.952122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.952081    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b"} err="failed to get container status \"d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b\": rpc error: code = NotFound desc = could not find container \"d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b\": container with ID starting with d15de5d5ee862656f6e9a4905c08a1f0e8b1250707cdffab05a7df5bf286b42b not found: ID does not exist"
Oct 31 19:39:25.952122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.952100    5551 scope.go:122] "RemoveContainer" containerID="94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315"
Oct 31 19:39:25.952338 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:39:25.952317    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315\": container with ID starting with 94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315 not found: ID does not exist" containerID="94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315"
Oct 31 19:39:25.952406 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:25.952348    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315"} err="failed to get container status \"94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315\": rpc error: code = NotFound desc = could not find container \"94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315\": container with ID starting with 94789850f4113e450a55137478135cd19737a301855bad3859b4c45d6d1a8315 not found: ID does not exist"
Oct 31 19:39:26.145674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:26.145641    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"]
Oct 31 19:39:26.149569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:26.149544    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bbd8c6-openshift-default-5fd44449f-rcgzj"]
Oct 31 19:39:26.911031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:26.910995    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" path="/var/lib/kubelet/pods/c3ff86d6-0892-42f9-b0b9-d3319a207fcb/volumes"
Oct 31 19:39:32.258548 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.258519    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"]
Oct 31 19:39:32.258965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.258724    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy" containerID="cri-o://242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724" gracePeriod=30
Oct 31 19:39:32.632604 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.632578    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"]
Oct 31 19:39:32.633121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633107    5551 cpu_manager.go:414] "containerMap: removing container" podUID="a777b590-d839-4db9-a4c4-8cb87403f3b7" containerName="maas-api"
Oct 31 19:39:32.633168 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633122    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a777b590-d839-4db9-a4c4-8cb87403f3b7" containerName="maas-api"
Oct 31 19:39:32.633168 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633141    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy"
Oct 31 19:39:32.633168 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633146    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy"
Oct 31 19:39:32.633266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633182    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy"
Oct 31 19:39:32.633266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633188    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy"
Oct 31 19:39:32.633266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633195    5551 cpu_manager.go:414] "containerMap: removing container" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy"
Oct 31 19:39:32.633266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633199    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy"
Oct 31 19:39:32.633392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633274    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e65d3d4-2f00-47ce-999d-3c13f5d95801" containerName="istio-proxy"
Oct 31 19:39:32.633392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633289    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d227addd-3b02-4f7d-aed0-a681fd8d680d" containerName="istio-proxy"
Oct 31 19:39:32.633392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633296    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy"
Oct 31 19:39:32.633392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.633302    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="a777b590-d839-4db9-a4c4-8cb87403f3b7" containerName="maas-api"
Oct 31 19:39:32.636684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.636668    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.684646 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.684612    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.684802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.684674    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.684871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.684805    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82sk4\" (UniqueName: \"kubernetes.io/projected/f757acab-a38f-4d08-a965-bf5e26f4ab6b-kube-api-access-82sk4\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.688980 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.688961    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:39:32.689116 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.689100    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:39:32.700421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.700394    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"]
Oct 31 19:39:32.785650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.785614    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.785830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.785679    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.785830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.785750    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82sk4\" (UniqueName: \"kubernetes.io/projected/f757acab-a38f-4d08-a965-bf5e26f4ab6b-kube-api-access-82sk4\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.786300 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.786277    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.788359 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.788328    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.842691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.842667    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82sk4\" (UniqueName: \"kubernetes.io/projected/f757acab-a38f-4d08-a965-bf5e26f4ab6b-kube-api-access-82sk4\") pod \"maas-api-revoke-sub-dfe7ed9e-4js9x\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:32.950024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:32.949946    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:33.173668 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.173622    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" containerID="cri-o://14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b" gracePeriod=45
Oct 31 19:39:33.189711 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.189670    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 31 19:39:33.334951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.334926    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"]
Oct 31 19:39:33.336493 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:39:33.336444    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf757acab_a38f_4d08_a965_bf5e26f4ab6b.slice/crio-e5ab2f301bee0b630a157578dd1251fa8d1ebe684b675f8003d7157266b4ab3e WatchSource:0}: Error finding container e5ab2f301bee0b630a157578dd1251fa8d1ebe684b675f8003d7157266b4ab3e: Status 404 returned error can't find the container with id e5ab2f301bee0b630a157578dd1251fa8d1ebe684b675f8003d7157266b4ab3e
Oct 31 19:39:33.432940 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.432915    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 19:39:33.495042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495005    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017d9396-474b-4678-b129-ca8e8b4e8643-kube-api-access-nlclc\" (UniqueName: \"kubernetes.io/projected/017d9396-474b-4678-b129-ca8e8b4e8643-kube-api-access-nlclc\") pod \"017d9396-474b-4678-b129-ca8e8b4e8643\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") "
Oct 31 19:39:33.495231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495101    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-tmp-dir\") pod \"017d9396-474b-4678-b129-ca8e8b4e8643\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") "
Oct 31 19:39:33.495231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495179    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-dshm\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-dshm\") pod \"017d9396-474b-4678-b129-ca8e8b4e8643\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") "
Oct 31 19:39:33.495231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495211    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-home\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-home\") pod \"017d9396-474b-4678-b129-ca8e8b4e8643\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") "
Oct 31 19:39:33.495420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495263    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/017d9396-474b-4678-b129-ca8e8b4e8643-tls-certs\" (UniqueName: \"kubernetes.io/secret/017d9396-474b-4678-b129-ca8e8b4e8643-tls-certs\") pod \"017d9396-474b-4678-b129-ca8e8b4e8643\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") "
Oct 31 19:39:33.495420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495391    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-model-cache\") pod \"017d9396-474b-4678-b129-ca8e8b4e8643\" (UID: \"017d9396-474b-4678-b129-ca8e8b4e8643\") "
Oct 31 19:39:33.496373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495856    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-tmp-dir" pod "017d9396-474b-4678-b129-ca8e8b4e8643" (UID: "017d9396-474b-4678-b129-ca8e8b4e8643"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:33.496373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495988    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-model-cache" pod "017d9396-474b-4678-b129-ca8e8b4e8643" (UID: "017d9396-474b-4678-b129-ca8e8b4e8643"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:33.496373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.495998    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-home" pod "017d9396-474b-4678-b129-ca8e8b4e8643" (UID: "017d9396-474b-4678-b129-ca8e8b4e8643"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:33.496373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.496158    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.496373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.496181    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.496373 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.496198    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.498481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.498436    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-dshm" pod "017d9396-474b-4678-b129-ca8e8b4e8643" (UID: "017d9396-474b-4678-b129-ca8e8b4e8643"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:33.498624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.498602    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017d9396-474b-4678-b129-ca8e8b4e8643-tls-certs" pod "017d9396-474b-4678-b129-ca8e8b4e8643" (UID: "017d9396-474b-4678-b129-ca8e8b4e8643"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:39:33.498703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.498647    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017d9396-474b-4678-b129-ca8e8b4e8643-kube-api-access-nlclc" pod "017d9396-474b-4678-b129-ca8e8b4e8643" (UID: "017d9396-474b-4678-b129-ca8e8b4e8643"). InnerVolumeSpecName "kube-api-access-nlclc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:33.597372 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.597333    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlclc\" (UniqueName: \"kubernetes.io/projected/017d9396-474b-4678-b129-ca8e8b4e8643-kube-api-access-nlclc\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.597372 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.597370    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017d9396-474b-4678-b129-ca8e8b4e8643-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.597641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.597385    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017d9396-474b-4678-b129-ca8e8b4e8643-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.996160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.996123    5551 generic.go:356] "Generic (PLEG): container finished" podID="017d9396-474b-4678-b129-ca8e8b4e8643" containerID="14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b" exitCode=0
Oct 31 19:39:33.996568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.996231    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" event={"ID":"017d9396-474b-4678-b129-ca8e8b4e8643","Type":"ContainerDied","Data":"14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b"}
Oct 31 19:39:33.996568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.996270    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp" event={"ID":"017d9396-474b-4678-b129-ca8e8b4e8643","Type":"ContainerDied","Data":"5923add63ae59bbf28de4a4177cc8d570059c0ad83fa5a538ae65792e72a54b2"}
Oct 31 19:39:33.996568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.996299    5551 scope.go:122] "RemoveContainer" containerID="14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b"
Oct 31 19:39:33.996568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:33.996425    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"
Oct 31 19:39:34.001978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:34.001947    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x" event={"ID":"f757acab-a38f-4d08-a965-bf5e26f4ab6b","Type":"ContainerStarted","Data":"66eff50b86bab034b39d95adcb98f62170d60408959b0ae0f8079ad074918145"}
Oct 31 19:39:34.002118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:34.002006    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x" event={"ID":"f757acab-a38f-4d08-a965-bf5e26f4ab6b","Type":"ContainerStarted","Data":"e5ab2f301bee0b630a157578dd1251fa8d1ebe684b675f8003d7157266b4ab3e"}
Oct 31 19:39:34.010170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:34.010152    5551 scope.go:122] "RemoveContainer" containerID="14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b"
Oct 31 19:39:34.010506 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:39:34.010481    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b\": container with ID starting with 14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b not found: ID does not exist" containerID="14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b"
Oct 31 19:39:34.010570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:34.010519    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b"} err="failed to get container status \"14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b\": rpc error: code = NotFound desc = could not find container \"14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b\": container with ID starting with 14cd7d1b9cb5e9d38a2c99013a0512a938df6186b7d6000e7f244599cfcf352b not found: ID does not exist"
Oct 31 19:39:34.164086 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:34.164055    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"]
Oct 31 19:39:34.168490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:34.168442    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/distinct-2-w0-a8587d-kserve-65f4855cd5-qbsgp"]
Oct 31 19:39:34.278854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:34.278764    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x" podStartSLOduration=2.27875094 podStartE2EDuration="2.27875094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:34.276706806 +0000 UTC m=+4677.507945098" watchObservedRunningTime="2026-10-31 19:39:34.27875094 +0000 UTC m=+4677.509989230"
Oct 31 19:39:34.909823 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:34.909788    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" path="/var/lib/kubelet/pods/017d9396-474b-4678-b129-ca8e8b4e8643/volumes"
Oct 31 19:39:35.011904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:35.011873    5551 generic.go:356] "Generic (PLEG): container finished" podID="f757acab-a38f-4d08-a965-bf5e26f4ab6b" containerID="66eff50b86bab034b39d95adcb98f62170d60408959b0ae0f8079ad074918145" exitCode=0
Oct 31 19:39:35.012127 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:35.011952    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x" event={"ID":"f757acab-a38f-4d08-a965-bf5e26f4ab6b","Type":"ContainerDied","Data":"66eff50b86bab034b39d95adcb98f62170d60408959b0ae0f8079ad074918145"}
Oct 31 19:39:36.193188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.193159    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:36.333272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.333170    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f757acab-a38f-4d08-a965-bf5e26f4ab6b-kube-api-access-82sk4\" (UniqueName: \"kubernetes.io/projected/f757acab-a38f-4d08-a965-bf5e26f4ab6b-kube-api-access-82sk4\") pod \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") "
Oct 31 19:39:36.333272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.333231    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-openshift-ca\") pod \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") "
Oct 31 19:39:36.333770 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.333739    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-openshift-ca" pod "f757acab-a38f-4d08-a965-bf5e26f4ab6b" (UID: "f757acab-a38f-4d08-a965-bf5e26f4ab6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:36.333864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.333817    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-xks-ca\") pod \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\" (UID: \"f757acab-a38f-4d08-a965-bf5e26f4ab6b\") "
Oct 31 19:39:36.334605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.334580    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:36.336293 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.336259    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-xks-ca" pod "f757acab-a38f-4d08-a965-bf5e26f4ab6b" (UID: "f757acab-a38f-4d08-a965-bf5e26f4ab6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:39:36.336677 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.336640    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f757acab-a38f-4d08-a965-bf5e26f4ab6b-kube-api-access-82sk4" pod "f757acab-a38f-4d08-a965-bf5e26f4ab6b" (UID: "f757acab-a38f-4d08-a965-bf5e26f4ab6b"). InnerVolumeSpecName "kube-api-access-82sk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:36.436763 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.436728    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f757acab-a38f-4d08-a965-bf5e26f4ab6b-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:36.436763 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:36.436764    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82sk4\" (UniqueName: \"kubernetes.io/projected/f757acab-a38f-4d08-a965-bf5e26f4ab6b-kube-api-access-82sk4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.034353 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.033412    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x" event={"ID":"f757acab-a38f-4d08-a965-bf5e26f4ab6b","Type":"ContainerDied","Data":"e5ab2f301bee0b630a157578dd1251fa8d1ebe684b675f8003d7157266b4ab3e"}
Oct 31 19:39:37.034353 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.033496    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ab2f301bee0b630a157578dd1251fa8d1ebe684b675f8003d7157266b4ab3e"
Oct 31 19:39:37.034353 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.033565    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"
Oct 31 19:39:37.554198 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.554134    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:39:37.653339 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653310    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-socket\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.653537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653377    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-data\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.653537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653421    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05089d3e-fe0e-47a2-b352-1625eb25fd09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05089d3e-fe0e-47a2-b352-1625eb25fd09-istiod-ca-cert\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.653537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653451    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-token\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-token\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.653537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653524    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-podinfo\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.653799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653568    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-certs\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.653799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653637    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-socket" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.653799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653674    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-kube-api-access-qn6xh\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-kube-api-access-qn6xh\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.653799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653703    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-credential-socket\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.653799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653784    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-envoy\") pod \"05089d3e-fe0e-47a2-b352-1625eb25fd09\" (UID: \"05089d3e-fe0e-47a2-b352-1625eb25fd09\") "
Oct 31 19:39:37.654093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653793    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-data" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.654093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653915    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-certs" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.654093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.653997    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-credential-socket" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.654352 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.654305    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05089d3e-fe0e-47a2-b352-1625eb25fd09-istiod-ca-cert" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:37.654583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.654495    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.654583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.654521    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.654583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.654536    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05089d3e-fe0e-47a2-b352-1625eb25fd09-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.654583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.654549    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.654583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.654562    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.656306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.656279    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-envoy" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.656306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.656287    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-podinfo" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:37.656717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.656695    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-kube-api-access-qn6xh" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "kube-api-access-qn6xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:37.656789 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.656724    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-token" pod "05089d3e-fe0e-47a2-b352-1625eb25fd09" (UID: "05089d3e-fe0e-47a2-b352-1625eb25fd09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:37.756201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.756162    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.756201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.756196    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.756201 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.756206    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05089d3e-fe0e-47a2-b352-1625eb25fd09-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.756448 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:37.756215    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn6xh\" (UniqueName: \"kubernetes.io/projected/05089d3e-fe0e-47a2-b352-1625eb25fd09-kube-api-access-qn6xh\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:38.047567 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.047531    5551 generic.go:356] "Generic (PLEG): container finished" podID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerID="242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724" exitCode=0
Oct 31 19:39:38.047748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.047593    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"
Oct 31 19:39:38.047748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.047614    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" event={"ID":"05089d3e-fe0e-47a2-b352-1625eb25fd09","Type":"ContainerDied","Data":"242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724"}
Oct 31 19:39:38.047748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.047681    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82" event={"ID":"05089d3e-fe0e-47a2-b352-1625eb25fd09","Type":"ContainerDied","Data":"9e059f6639332a0d247610114fd59095a6dbd72a9b71e962fed1dc5ee1e4c476"}
Oct 31 19:39:38.047748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.047711    5551 scope.go:122] "RemoveContainer" containerID="242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724"
Oct 31 19:39:38.068051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.068010    5551 scope.go:122] "RemoveContainer" containerID="242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724"
Oct 31 19:39:38.068587 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:39:38.068564    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724\": container with ID starting with 242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724 not found: ID does not exist" containerID="242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724"
Oct 31 19:39:38.068686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.068597    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724"} err="failed to get container status \"242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724\": rpc error: code = NotFound desc = could not find container \"242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724\": container with ID starting with 242c5badc3a87c8573f543a321606dcb2bc4f1b05ce80e7e2f0a2129b7b4b724 not found: ID does not exist"
Oct 31 19:39:38.218032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.218000    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"]
Oct 31 19:39:38.269144 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.269109    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bkh82"]
Oct 31 19:39:38.910252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:38.910213    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" path="/var/lib/kubelet/pods/05089d3e-fe0e-47a2-b352-1625eb25fd09/volumes"
Oct 31 19:39:45.332805 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:45.332715    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"]
Oct 31 19:39:46.608740 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:39:46.608706    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:39:50.786056 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:50.786018    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"]
Oct 31 19:39:58.901425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.901381    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"]
Oct 31 19:39:58.902217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902192    5551 cpu_manager.go:414] "containerMap: removing container" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy"
Oct 31 19:39:58.902325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902219    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy"
Oct 31 19:39:58.902325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902241    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f757acab-a38f-4d08-a965-bf5e26f4ab6b" containerName="revoke-keys"
Oct 31 19:39:58.902325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902248    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f757acab-a38f-4d08-a965-bf5e26f4ab6b" containerName="revoke-keys"
Oct 31 19:39:58.902325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902275    5551 cpu_manager.go:414] "containerMap: removing container" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main"
Oct 31 19:39:58.902325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902282    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main"
Oct 31 19:39:58.902633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902513    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="05089d3e-fe0e-47a2-b352-1625eb25fd09" containerName="istio-proxy"
Oct 31 19:39:58.902633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902555    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ff86d6-0892-42f9-b0b9-d3319a207fcb" containerName="istio-proxy"
Oct 31 19:39:58.902633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902570    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f757acab-a38f-4d08-a965-bf5e26f4ab6b" containerName="revoke-keys"
Oct 31 19:39:58.902633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.902585    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="017d9396-474b-4678-b129-ca8e8b4e8643" containerName="main"
Oct 31 19:39:58.906058 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.906024    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:58.958504 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:58.958449    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a8587d-openshift-default-dockercfg-zdp6b\""
Oct 31 19:39:59.012767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.012735    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"]
Oct 31 19:39:59.032140 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032105    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2s9\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-kube-api-access-4t2s9\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.032281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032237    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1507e13-825a-46b7-97bd-dbf780d549c3-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.032487 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032425    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.032640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032524    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.032714 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032682    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.032779 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032722    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.032841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032807    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.032902 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032843    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.032965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.032898    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134067 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134029    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2s9\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-kube-api-access-4t2s9\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134092    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1507e13-825a-46b7-97bd-dbf780d549c3-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134173    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134197    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134252 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134248    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134275    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134330    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134368    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134395    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134750 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134718    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134729    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134783    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.134927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.134885    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1507e13-825a-46b7-97bd-dbf780d549c3-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.135129 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.135105    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.136706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.136680    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.136945 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.136925    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.192104 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.192034    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.192500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.192449    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2s9\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-kube-api-access-4t2s9\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.221306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.221280    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:39:59.663433 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.663407    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"]
Oct 31 19:39:59.665641 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:39:59.665610    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1507e13_825a_46b7_97bd_dbf780d549c3.slice/crio-097b103e0621d591347a4881b2a85a6e2edc4b9f27e3b318a345a3b54b8bd6a6 WatchSource:0}: Error finding container 097b103e0621d591347a4881b2a85a6e2edc4b9f27e3b318a345a3b54b8bd6a6: Status 404 returned error can't find the container with id 097b103e0621d591347a4881b2a85a6e2edc4b9f27e3b318a345a3b54b8bd6a6
Oct 31 19:39:59.668444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.668407    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:39:59.668566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.668515    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:39:59.668566 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:39:59.668549    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:40:00.204216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.204178    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" event={"ID":"d1507e13-825a-46b7-97bd-dbf780d549c3","Type":"ContainerStarted","Data":"2f87b521858d7178be2941bc14fb46adc226a7a9b275843acdc3f5072d57d414"}
Oct 31 19:40:00.204785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.204255    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" event={"ID":"d1507e13-825a-46b7-97bd-dbf780d549c3","Type":"ContainerStarted","Data":"097b103e0621d591347a4881b2a85a6e2edc4b9f27e3b318a345a3b54b8bd6a6"}
Oct 31 19:40:00.221624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.221596    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:40:00.222993 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.222967    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 31 19:40:00.223143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.223020    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 31 19:40:00.332546 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.332486    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podStartSLOduration=2.332470784 podStartE2EDuration="2.332470784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:00.327490687 +0000 UTC m=+4703.558728975" watchObservedRunningTime="2026-10-31 19:40:00.332470784 +0000 UTC m=+4703.563709066"
Oct 31 19:40:00.372483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.371655    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb" podUID="86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" containerName="main" containerID="cri-o://a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516" gracePeriod=45
Oct 31 19:40:00.717105 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.717072    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:40:00.880928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.880896    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-dshm\") pod \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") "
Oct 31 19:40:00.881213 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.880952    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-model-cache\") pod \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") "
Oct 31 19:40:00.881213 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.881023    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tmp-dir\") pod \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") "
Oct 31 19:40:00.881213 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.881077    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tls-certs\" (UniqueName: \"kubernetes.io/secret/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tls-certs\") pod \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") "
Oct 31 19:40:00.881213 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.881201    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-kube-api-access-jn5p4\" (UniqueName: \"kubernetes.io/projected/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-kube-api-access-jn5p4\") pod \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") "
Oct 31 19:40:00.881494 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.881231    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-home\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-home\") pod \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\" (UID: \"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e\") "
Oct 31 19:40:00.881990 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.881770    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tmp-dir" pod "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" (UID: "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:00.882122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.881989    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-home" pod "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" (UID: "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:00.882122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.882023    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-model-cache" pod "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" (UID: "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:00.884248 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.884226    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-kube-api-access-jn5p4" pod "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" (UID: "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e"). InnerVolumeSpecName "kube-api-access-jn5p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:00.884779 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.884764    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-dshm" pod "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" (UID: "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:00.884863 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.884784    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tls-certs" pod "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" (UID: "86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:00.983073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.983026    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn5p4\" (UniqueName: \"kubernetes.io/projected/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-kube-api-access-jn5p4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:00.983073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.983060    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:00.983073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.983074    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:00.983335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.983085    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:00.983335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.983101    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:00.983335 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:00.983115    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:01.212778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.212693    5551 generic.go:356] "Generic (PLEG): container finished" podID="86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" containerID="a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516" exitCode=0
Oct 31 19:40:01.212778 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.212742    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"
Oct 31 19:40:01.213315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.212777    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb" event={"ID":"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e","Type":"ContainerDied","Data":"a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516"}
Oct 31 19:40:01.213315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.212829    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb" event={"ID":"86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e","Type":"ContainerDied","Data":"e0800a69b901009a6c303ba1c4d213fc3f59614b29c4f70a8ef4904d1a5dea5f"}
Oct 31 19:40:01.213315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.212851    5551 scope.go:122] "RemoveContainer" containerID="a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516"
Oct 31 19:40:01.222777 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.222751    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 31 19:40:01.222904 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.222802    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 31 19:40:01.223602 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.223587    5551 scope.go:122] "RemoveContainer" containerID="a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516"
Oct 31 19:40:01.223864 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:40:01.223847    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516\": container with ID starting with a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516 not found: ID does not exist" containerID="a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516"
Oct 31 19:40:01.223928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.223872    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516"} err="failed to get container status \"a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516\": rpc error: code = NotFound desc = could not find container \"a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516\": container with ID starting with a49c5d3655e1b165af4201232f4d8800db84522e1da9fa976d3986e785766516 not found: ID does not exist"
Oct 31 19:40:01.385565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.385532    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"]
Oct 31 19:40:01.393550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.393522    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb"]
Oct 31 19:40:01.595713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:01.595611    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-d1ff32/rate-test-model-d1ff32-kserve-64b6bf55b5-x9qkb" podUID="86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.222:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:02.221629 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:02.221596    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 31 19:40:02.222045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:02.221646    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 31 19:40:02.908797 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:02.908763    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" path="/var/lib/kubelet/pods/86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e/volumes"
Oct 31 19:40:03.222655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:03.222557    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 31 19:40:03.222655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:03.222624    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 31 19:40:04.221678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:04.221638    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 31 19:40:04.221901 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:04.221732    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 31 19:40:05.222686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:05.222648    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 31 19:40:05.223158 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:05.222725    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 31 19:40:05.817215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:05.816941    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p" podUID="d94ebace-d451-48fa-81c6-d9970c417721" containerName="main" containerID="cri-o://00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36" gracePeriod=45
Oct 31 19:40:06.171016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.170988    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:40:06.222518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.222480    5551 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 31 19:40:06.222720 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.222536    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 31 19:40:06.254206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.254172    5551 generic.go:356] "Generic (PLEG): container finished" podID="d94ebace-d451-48fa-81c6-d9970c417721" containerID="00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36" exitCode=0
Oct 31 19:40:06.254528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.254234    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"
Oct 31 19:40:06.254622 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.254261    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p" event={"ID":"d94ebace-d451-48fa-81c6-d9970c417721","Type":"ContainerDied","Data":"00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36"}
Oct 31 19:40:06.254685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.254647    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p" event={"ID":"d94ebace-d451-48fa-81c6-d9970c417721","Type":"ContainerDied","Data":"76af632035008ccdcfdd4b798b20633b9422b89362c14c12199d57c2c30b698f"}
Oct 31 19:40:06.254685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.254678    5551 scope.go:122] "RemoveContainer" containerID="00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36"
Oct 31 19:40:06.268606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.268584    5551 scope.go:122] "RemoveContainer" containerID="00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36"
Oct 31 19:40:06.268952 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:40:06.268929    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36\": container with ID starting with 00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36 not found: ID does not exist" containerID="00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36"
Oct 31 19:40:06.269037 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.268962    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36"} err="failed to get container status \"00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36\": rpc error: code = NotFound desc = could not find container \"00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36\": container with ID starting with 00097689cc5277f13f4981e87ccf451c62c830523adc688934a2fd05d11e3c36 not found: ID does not exist"
Oct 31 19:40:06.349036 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349001    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-home\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-home\") pod \"d94ebace-d451-48fa-81c6-d9970c417721\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") "
Oct 31 19:40:06.349279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349060    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94ebace-d451-48fa-81c6-d9970c417721-kube-api-access-8tzwt\" (UniqueName: \"kubernetes.io/projected/d94ebace-d451-48fa-81c6-d9970c417721-kube-api-access-8tzwt\") pod \"d94ebace-d451-48fa-81c6-d9970c417721\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") "
Oct 31 19:40:06.349279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349237    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-tmp-dir\") pod \"d94ebace-d451-48fa-81c6-d9970c417721\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") "
Oct 31 19:40:06.349433 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349289    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-model-cache\") pod \"d94ebace-d451-48fa-81c6-d9970c417721\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") "
Oct 31 19:40:06.349614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349313    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-home" pod "d94ebace-d451-48fa-81c6-d9970c417721" (UID: "d94ebace-d451-48fa-81c6-d9970c417721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:06.349684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349610    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-tmp-dir" pod "d94ebace-d451-48fa-81c6-d9970c417721" (UID: "d94ebace-d451-48fa-81c6-d9970c417721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:06.349684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349623    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-model-cache" pod "d94ebace-d451-48fa-81c6-d9970c417721" (UID: "d94ebace-d451-48fa-81c6-d9970c417721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:06.349799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349754    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-dshm\") pod \"d94ebace-d451-48fa-81c6-d9970c417721\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") "
Oct 31 19:40:06.349862 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.349835    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d94ebace-d451-48fa-81c6-d9970c417721-tls-certs\" (UniqueName: \"kubernetes.io/secret/d94ebace-d451-48fa-81c6-d9970c417721-tls-certs\") pod \"d94ebace-d451-48fa-81c6-d9970c417721\" (UID: \"d94ebace-d451-48fa-81c6-d9970c417721\") "
Oct 31 19:40:06.350665 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.350602    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:06.350792 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.350673    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:06.350792 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.350689    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:06.353328 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.353288    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d94ebace-d451-48fa-81c6-d9970c417721-tls-certs" pod "d94ebace-d451-48fa-81c6-d9970c417721" (UID: "d94ebace-d451-48fa-81c6-d9970c417721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:06.366538 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.366513    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94ebace-d451-48fa-81c6-d9970c417721-kube-api-access-8tzwt" pod "d94ebace-d451-48fa-81c6-d9970c417721" (UID: "d94ebace-d451-48fa-81c6-d9970c417721"). InnerVolumeSpecName "kube-api-access-8tzwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:06.367218 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.367183    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-dshm" pod "d94ebace-d451-48fa-81c6-d9970c417721" (UID: "d94ebace-d451-48fa-81c6-d9970c417721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:06.451924 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.451895    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tzwt\" (UniqueName: \"kubernetes.io/projected/d94ebace-d451-48fa-81c6-d9970c417721-kube-api-access-8tzwt\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:06.452052 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.451929    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94ebace-d451-48fa-81c6-d9970c417721-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:06.452052 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.451944    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94ebace-d451-48fa-81c6-d9970c417721-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:06.717810 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.717707    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"]
Oct 31 19:40:06.720580 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.720550    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1aefcd/rate-test-model-1aefcd-kserve-679fd5f9f7-lbj8p"]
Oct 31 19:40:06.911478 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:06.911435    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94ebace-d451-48fa-81c6-d9970c417721" path="/var/lib/kubelet/pods/d94ebace-d451-48fa-81c6-d9970c417721/volumes"
Oct 31 19:40:07.196484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.196225    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"]
Oct 31 19:40:07.197013 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.196975    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d94ebace-d451-48fa-81c6-d9970c417721" containerName="main"
Oct 31 19:40:07.197330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.197016    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94ebace-d451-48fa-81c6-d9970c417721" containerName="main"
Oct 31 19:40:07.197330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.197069    5551 cpu_manager.go:414] "containerMap: removing container" podUID="86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" containerName="main"
Oct 31 19:40:07.197330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.197082    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" containerName="main"
Oct 31 19:40:07.197330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.197246    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94ebace-d451-48fa-81c6-d9970c417721" containerName="main"
Oct 31 19:40:07.197330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.197266    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ae8e0e-0b8a-44a7-a5f4-4a5b4661ea9e" containerName="main"
Oct 31 19:40:07.202250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.202222    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.227087 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.227040    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:40:07.227611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.227497    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:40:07.228308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.228286    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:40:07.258049 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.257988    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-0f0e4e-serving-cert\""
Oct 31 19:40:07.261985 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.261948    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"]
Oct 31 19:40:07.364854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.363057    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.364854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.363264    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555fbf73-7e06-47c8-86ac-4e2236813469-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.364854 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.363301    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.374079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.374042    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrzd\" (UniqueName: \"kubernetes.io/projected/555fbf73-7e06-47c8-86ac-4e2236813469-kube-api-access-mjrzd\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.475729 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.475637    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.475874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.475741    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555fbf73-7e06-47c8-86ac-4e2236813469-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.475874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.475769    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.475963 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.475886    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrzd\" (UniqueName: \"kubernetes.io/projected/555fbf73-7e06-47c8-86ac-4e2236813469-kube-api-access-mjrzd\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.476248 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:40:07.476231    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-serving-cert: secret "maas-api-e2e-shared-b-w1-0f0e4e-serving-cert" not found
Oct 31 19:40:07.476305 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:40:07.476296    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls podName:555fbf73-7e06-47c8-86ac-4e2236813469 nodeName:}" failed. No retries permitted until 2026-10-31 19:40:07.976280469 +0000 UTC m=+4711.207518749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls") pod "maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx" (UID: "555fbf73-7e06-47c8-86ac-4e2236813469") : secret "maas-api-e2e-shared-b-w1-0f0e4e-serving-cert" not found
Oct 31 19:40:07.476882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.476859    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555fbf73-7e06-47c8-86ac-4e2236813469-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.479732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.479706    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.533754 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.533717    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrzd\" (UniqueName: \"kubernetes.io/projected/555fbf73-7e06-47c8-86ac-4e2236813469-kube-api-access-mjrzd\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.982830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.982794    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:07.985689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:07.985660    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:08.118892 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:08.118860    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:08.504812 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:08.504785    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"]
Oct 31 19:40:08.506111 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:40:08.506083    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555fbf73_7e06_47c8_86ac_4e2236813469.slice/crio-793745d6cd19bd011c9a56c6a21277f1ec4a5789335862e21668e5c73234b006 WatchSource:0}: Error finding container 793745d6cd19bd011c9a56c6a21277f1ec4a5789335862e21668e5c73234b006: Status 404 returned error can't find the container with id 793745d6cd19bd011c9a56c6a21277f1ec4a5789335862e21668e5c73234b006
Oct 31 19:40:08.507636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:08.507615    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:40:09.283543 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:09.283499    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx" event={"ID":"555fbf73-7e06-47c8-86ac-4e2236813469","Type":"ContainerStarted","Data":"793745d6cd19bd011c9a56c6a21277f1ec4a5789335862e21668e5c73234b006"}
Oct 31 19:40:10.292303 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:10.292262    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx" event={"ID":"555fbf73-7e06-47c8-86ac-4e2236813469","Type":"ContainerStarted","Data":"e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977"}
Oct 31 19:40:10.292767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:10.292333    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:10.424498 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:10.421747    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx" podStartSLOduration=2.515669332 podStartE2EDuration="3.421728167s" totalImagesPullingTime="906.058835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:10.41174032 +0000 UTC m=+4713.642978611" watchObservedRunningTime="2026-10-31 19:40:10.421728167 +0000 UTC m=+4713.652966458"
Oct 31 19:40:16.304265 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:16.304231    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:40:16.604945 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:40:16.604858    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:40:27.825549 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:27.825515    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"]
Oct 31 19:40:27.826027 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:27.825735    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy" containerID="cri-o://399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9" gracePeriod=30
Oct 31 19:40:29.199072 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:29.199034    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"]
Oct 31 19:40:29.199567 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:29.199290    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy" containerID="cri-o://2f87b521858d7178be2941bc14fb46adc226a7a9b275843acdc3f5072d57d414" gracePeriod=30
Oct 31 19:40:31.041312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:31.041279    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:40:31.041733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:31.041330    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:40:33.203413 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.203388    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:40:33.279859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.279278    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32f2799-c8ee-418d-ba27-31d90cba3a40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f2799-c8ee-418d-ba27-31d90cba3a40-istiod-ca-cert\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.279859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.279337    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-kube-api-access-mh6h9\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-kube-api-access-mh6h9\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.279859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.279441    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-envoy\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.280674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.280627    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32f2799-c8ee-418d-ba27-31d90cba3a40-istiod-ca-cert" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:33.280674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.280666    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-token\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-token\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.280869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.280730    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-socket\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.280869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.280771    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-data\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.280869 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.280832    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-podinfo\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.281038 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.280874    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-certs\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.281038 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.280931    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-credential-socket\") pod \"d32f2799-c8ee-418d-ba27-31d90cba3a40\" (UID: \"d32f2799-c8ee-418d-ba27-31d90cba3a40\") "
Oct 31 19:40:33.281452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.281425    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-data" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:33.281650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.281629    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32f2799-c8ee-418d-ba27-31d90cba3a40-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.281976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.281952    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-certs" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:33.282136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.281963    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-socket" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:33.282136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.282085    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-credential-socket" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:33.283045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.283000    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-envoy" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:33.283152 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.283121    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-kube-api-access-mh6h9" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "kube-api-access-mh6h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:33.284623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.284597    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-podinfo" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:33.284841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.284814    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-token" pod "d32f2799-c8ee-418d-ba27-31d90cba3a40" (UID: "d32f2799-c8ee-418d-ba27-31d90cba3a40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:33.390841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.390364    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6h9\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-kube-api-access-mh6h9\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.390841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.390405    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.390841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.390422    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.390841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.390436    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.390841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.390450    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.390841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.390511    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32f2799-c8ee-418d-ba27-31d90cba3a40-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.390841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.390525    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.390841 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.390538    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32f2799-c8ee-418d-ba27-31d90cba3a40-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:33.465533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.461924    5551 generic.go:356] "Generic (PLEG): container finished" podID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerID="399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9" exitCode=0
Oct 31 19:40:33.465533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.461991    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" event={"ID":"d32f2799-c8ee-418d-ba27-31d90cba3a40","Type":"ContainerDied","Data":"399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9"}
Oct 31 19:40:33.465533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.462044    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq" event={"ID":"d32f2799-c8ee-418d-ba27-31d90cba3a40","Type":"ContainerDied","Data":"55897f8a4f653fd4ace1249a50e6deee3ad0ebc7e66d57b13695a4a64333651f"}
Oct 31 19:40:33.465533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.462075    5551 scope.go:122] "RemoveContainer" containerID="399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9"
Oct 31 19:40:33.465533 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.462240    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"
Oct 31 19:40:33.491738 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.491707    5551 scope.go:122] "RemoveContainer" containerID="e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0"
Oct 31 19:40:33.501591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.501569    5551 scope.go:122] "RemoveContainer" containerID="399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9"
Oct 31 19:40:33.501898 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:40:33.501877    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9\": container with ID starting with 399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9 not found: ID does not exist" containerID="399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9"
Oct 31 19:40:33.501986 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.501908    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9"} err="failed to get container status \"399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9\": rpc error: code = NotFound desc = could not find container \"399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9\": container with ID starting with 399108a7530844e83fe8249212dfcbea227d68d8b2d6c2cd29fb85509e7d89a9 not found: ID does not exist"
Oct 31 19:40:33.501986 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.501929    5551 scope.go:122] "RemoveContainer" containerID="e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0"
Oct 31 19:40:33.502217 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:40:33.502199    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0\": container with ID starting with e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0 not found: ID does not exist" containerID="e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0"
Oct 31 19:40:33.502266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.502229    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0"} err="failed to get container status \"e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0\": rpc error: code = NotFound desc = could not find container \"e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0\": container with ID starting with e2dba9d6b01b78c82e95c068f5c8175df6f303c9981b41396d49b07c95eea8c0 not found: ID does not exist"
Oct 31 19:40:33.635388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.635351    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"]
Oct 31 19:40:33.640145 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:33.640109    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d22e36-openshift-default-dbcdf9b87-584rq"]
Oct 31 19:40:34.496738 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.496694    5551 generic.go:356] "Generic (PLEG): container finished" podID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerID="2f87b521858d7178be2941bc14fb46adc226a7a9b275843acdc3f5072d57d414" exitCode=0
Oct 31 19:40:34.497182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.496842    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" event={"ID":"d1507e13-825a-46b7-97bd-dbf780d549c3","Type":"ContainerDied","Data":"2f87b521858d7178be2941bc14fb46adc226a7a9b275843acdc3f5072d57d414"}
Oct 31 19:40:34.523963 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.523934    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:40:34.605760 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.605718    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-credential-socket\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.605962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.605768    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-socket\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.605962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.605833    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-podinfo\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.605962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.605896    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-envoy\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.605962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.605958    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1507e13-825a-46b7-97bd-dbf780d549c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1507e13-825a-46b7-97bd-dbf780d549c3-istiod-ca-cert\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.606194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.605999    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-kube-api-access-4t2s9\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-kube-api-access-4t2s9\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.606194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606026    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-credential-socket" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.606194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606036    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-socket" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.606194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606039    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-token\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-token\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.606194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606098    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-data\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.606194 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606161    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-certs\") pod \"d1507e13-825a-46b7-97bd-dbf780d549c3\" (UID: \"d1507e13-825a-46b7-97bd-dbf780d549c3\") "
Oct 31 19:40:34.606601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606321    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-data" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.606601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606335    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1507e13-825a-46b7-97bd-dbf780d549c3-istiod-ca-cert" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:34.606601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606501    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-certs" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.606946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606731    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1507e13-825a-46b7-97bd-dbf780d549c3-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.606946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606747    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.606946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606760    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.606946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606775    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.606946 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.606789    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.608521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.608494    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-envoy" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.608665 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.608529    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-token" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:34.608956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.608916    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-podinfo" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:34.608956 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.608924    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-kube-api-access-4t2s9" pod "d1507e13-825a-46b7-97bd-dbf780d549c3" (UID: "d1507e13-825a-46b7-97bd-dbf780d549c3"). InnerVolumeSpecName "kube-api-access-4t2s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:34.708063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.708025    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.708063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.708066    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.708264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.708083    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2s9\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-kube-api-access-4t2s9\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.708264 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.708099    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1507e13-825a-46b7-97bd-dbf780d549c3-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.910918 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:34.910884    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" path="/var/lib/kubelet/pods/d32f2799-c8ee-418d-ba27-31d90cba3a40/volumes"
Oct 31 19:40:35.508717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:35.508689    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"
Oct 31 19:40:35.509114 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:35.508708    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn" event={"ID":"d1507e13-825a-46b7-97bd-dbf780d549c3","Type":"ContainerDied","Data":"097b103e0621d591347a4881b2a85a6e2edc4b9f27e3b318a345a3b54b8bd6a6"}
Oct 31 19:40:35.509114 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:35.508776    5551 scope.go:122] "RemoveContainer" containerID="2f87b521858d7178be2941bc14fb46adc226a7a9b275843acdc3f5072d57d414"
Oct 31 19:40:35.730674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:35.730639    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"]
Oct 31 19:40:35.784175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:35.784087    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-95tcn"]
Oct 31 19:40:36.910779 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:36.910743    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" path="/var/lib/kubelet/pods/d1507e13-825a-46b7-97bd-dbf780d549c3/volumes"
Oct 31 19:40:40.792641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:40.792598    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:40:40.793079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:40.792689    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:42.576340 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:42.576145    5551 generic.go:356] "Generic (PLEG): container finished" podID="611474c7-427f-438e-83ed-257b09db4ff1" containerID="1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97" exitCode=0
Oct 31 19:40:42.576960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:42.576382    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" event={"ID":"611474c7-427f-438e-83ed-257b09db4ff1","Type":"ContainerDied","Data":"1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97"}
Oct 31 19:40:42.576960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:42.576868    5551 scope.go:122] "RemoveContainer" containerID="1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97"
Oct 31 19:40:42.579389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:42.579346    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:40:42.579565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:42.579443    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:40:42.579565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:42.579513    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:40:43.585064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:43.585028    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" event={"ID":"611474c7-427f-438e-83ed-257b09db4ff1","Type":"ContainerStarted","Data":"c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3"}
Oct 31 19:40:43.967500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:43.966128    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:40:43.967500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:43.966717    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 19:40:43.967500 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:43.966763    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 19:40:44.974831 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:44.974792    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 19:40:44.975338 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:44.974848    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 19:40:45.971255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:45.970639    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 19:40:45.971649 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:45.971595    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 19:40:46.606385 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:40:46.606345    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:40:46.969493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:46.967106    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 19:40:46.969493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:46.967204    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 19:40:47.965484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:47.961865    5551 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 19:40:47.965484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:47.961946    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 19:40:48.962090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:48.962055    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:40:48.962611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:48.962583    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:40:48.963506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:40:48.963424    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:41:03.503723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:03.503678    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"]
Oct 31 19:41:03.504199 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:03.503913    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy" containerID="cri-o://ba067a5aa0df8a18109e70562c6ff08f223f9da123147b4686761ad85be6318d" gracePeriod=30
Oct 31 19:41:04.297255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:04.297224    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:41:04.297440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:04.297276    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:41:08.758609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.758577    5551 generic.go:356] "Generic (PLEG): container finished" podID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerID="ba067a5aa0df8a18109e70562c6ff08f223f9da123147b4686761ad85be6318d" exitCode=0
Oct 31 19:41:08.758984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.758657    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" event={"ID":"7ebec53a-ceec-4521-8d79-b345aeefef4f","Type":"ContainerDied","Data":"ba067a5aa0df8a18109e70562c6ff08f223f9da123147b4686761ad85be6318d"}
Oct 31 19:41:08.758984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.758715    5551 scope.go:122] "RemoveContainer" containerID="a79a0ed3f29b8343fe527ea0fa718b453c5ba51ccbe4011055dabf4f78cae591"
Oct 31 19:41:08.777608 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.777590    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:41:08.857214 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857182    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-data\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857340 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857238    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-kube-api-access-ssfcj\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-kube-api-access-ssfcj\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857340 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857285    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-socket\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857337    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-certs\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857387    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-envoy\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857435    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-podinfo\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857487    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-data" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:08.857658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857500    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebec53a-ceec-4521-8d79-b345aeefef4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebec53a-ceec-4521-8d79-b345aeefef4f-istiod-ca-cert\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857545    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-token\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-token\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857583    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-credential-socket\") pod \"7ebec53a-ceec-4521-8d79-b345aeefef4f\" (UID: \"7ebec53a-ceec-4521-8d79-b345aeefef4f\") "
Oct 31 19:41:08.857897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857694    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-certs" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:08.857897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857726    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-socket" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:08.858008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.857972    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-credential-socket" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:08.858121 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.858104    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:08.858187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.858127    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:08.858187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.858142    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:08.858187 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.858175    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:08.858311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.858183    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebec53a-ceec-4521-8d79-b345aeefef4f-istiod-ca-cert" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:08.859730 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.859706    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-podinfo" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:08.859827 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.859801    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-token" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:08.859964 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.859949    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-kube-api-access-ssfcj" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "kube-api-access-ssfcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:08.860303 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.860285    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-envoy" pod "7ebec53a-ceec-4521-8d79-b345aeefef4f" (UID: "7ebec53a-ceec-4521-8d79-b345aeefef4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:08.959906 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.959874    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssfcj\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-kube-api-access-ssfcj\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:08.959906 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.959905    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:08.960118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.959919    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:08.960118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.959933    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebec53a-ceec-4521-8d79-b345aeefef4f-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:08.960118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:08.959945    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebec53a-ceec-4521-8d79-b345aeefef4f-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:09.766391 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:09.766360    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"
Oct 31 19:41:09.766897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:09.766402    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784" event={"ID":"7ebec53a-ceec-4521-8d79-b345aeefef4f","Type":"ContainerDied","Data":"6f99cdd4ff4bc0593629b91aa94ce94e5715371e4123ce406f7f13dc65e032cb"}
Oct 31 19:41:09.766897 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:09.766469    5551 scope.go:122] "RemoveContainer" containerID="ba067a5aa0df8a18109e70562c6ff08f223f9da123147b4686761ad85be6318d"
Oct 31 19:41:10.058127 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:10.058048    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"]
Oct 31 19:41:10.060817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:10.060784    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d1ff32-openshift-default-74b9786d5f-v2784"]
Oct 31 19:41:10.913219 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:10.913176    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" path="/var/lib/kubelet/pods/7ebec53a-ceec-4521-8d79-b345aeefef4f/volumes"
Oct 31 19:41:16.606470 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:41:16.606370    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:41:24.604840 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.604797    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"]
Oct 31 19:41:24.605282 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.605022    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n" podUID="4429158d-d586-4a08-81bf-ba74734f82fc" containerName="maas-api" containerID="cri-o://4c409f2e39942ff013fcf709116aaa58f280c3dc6efdffce99c267739ae95ba5" gracePeriod=30
Oct 31 19:41:24.876417 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.876387    5551 generic.go:356] "Generic (PLEG): container finished" podID="4429158d-d586-4a08-81bf-ba74734f82fc" containerID="4c409f2e39942ff013fcf709116aaa58f280c3dc6efdffce99c267739ae95ba5" exitCode=0
Oct 31 19:41:24.876595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.876443    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n" event={"ID":"4429158d-d586-4a08-81bf-ba74734f82fc","Type":"ContainerDied","Data":"4c409f2e39942ff013fcf709116aaa58f280c3dc6efdffce99c267739ae95ba5"}
Oct 31 19:41:24.876595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.876513    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n" event={"ID":"4429158d-d586-4a08-81bf-ba74734f82fc","Type":"ContainerDied","Data":"6cc2da63ff2be4ca612518bcb3fe423239d667699b72c919554439595b1a83b4"}
Oct 31 19:41:24.876595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.876538    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cc2da63ff2be4ca612518bcb3fe423239d667699b72c919554439595b1a83b4"
Oct 31 19:41:24.884261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.884236    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 19:41:24.970136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.970107    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-metrics-certs\") pod \"4429158d-d586-4a08-81bf-ba74734f82fc\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") "
Oct 31 19:41:24.970330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.970177    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls\") pod \"4429158d-d586-4a08-81bf-ba74734f82fc\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") "
Oct 31 19:41:24.970330 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.970303    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4429158d-d586-4a08-81bf-ba74734f82fc-kube-api-access-bzg5v\" (UniqueName: \"kubernetes.io/projected/4429158d-d586-4a08-81bf-ba74734f82fc-kube-api-access-bzg5v\") pod \"4429158d-d586-4a08-81bf-ba74734f82fc\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") "
Oct 31 19:41:24.970518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.970405    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4429158d-d586-4a08-81bf-ba74734f82fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4429158d-d586-4a08-81bf-ba74734f82fc-openshift-service-ca\") pod \"4429158d-d586-4a08-81bf-ba74734f82fc\" (UID: \"4429158d-d586-4a08-81bf-ba74734f82fc\") "
Oct 31 19:41:24.970998 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.970967    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4429158d-d586-4a08-81bf-ba74734f82fc-openshift-service-ca" pod "4429158d-d586-4a08-81bf-ba74734f82fc" (UID: "4429158d-d586-4a08-81bf-ba74734f82fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:24.972894 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.972867    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4429158d-d586-4a08-81bf-ba74734f82fc-kube-api-access-bzg5v" pod "4429158d-d586-4a08-81bf-ba74734f82fc" (UID: "4429158d-d586-4a08-81bf-ba74734f82fc"). InnerVolumeSpecName "kube-api-access-bzg5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:24.973045 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.973025    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls" pod "4429158d-d586-4a08-81bf-ba74734f82fc" (UID: "4429158d-d586-4a08-81bf-ba74734f82fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:41:24.973188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:24.973163    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-metrics-certs" pod "4429158d-d586-4a08-81bf-ba74734f82fc" (UID: "4429158d-d586-4a08-81bf-ba74734f82fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:41:25.072031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:25.071984    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzg5v\" (UniqueName: \"kubernetes.io/projected/4429158d-d586-4a08-81bf-ba74734f82fc-kube-api-access-bzg5v\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.072031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:25.072026    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4429158d-d586-4a08-81bf-ba74734f82fc-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.072258 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:25.072041    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.072258 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:25.072055    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4429158d-d586-4a08-81bf-ba74734f82fc-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.884676 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:25.884637    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"
Oct 31 19:41:26.048350 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:26.048302    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"]
Oct 31 19:41:26.097793 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:26.097761    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3e4a2c-858d587f79-qtv6n"]
Oct 31 19:41:26.908095 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:26.908064    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4429158d-d586-4a08-81bf-ba74734f82fc" path="/var/lib/kubelet/pods/4429158d-d586-4a08-81bf-ba74734f82fc/volumes"
Oct 31 19:41:29.557070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557025    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"]
Oct 31 19:41:29.557592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557575    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557593    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557611    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4429158d-d586-4a08-81bf-ba74734f82fc" containerName="maas-api"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557618    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4429158d-d586-4a08-81bf-ba74734f82fc" containerName="maas-api"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557638    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557644    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557657    5551 cpu_manager.go:414] "containerMap: removing container" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557662    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557672    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy"
Oct 31 19:41:29.557679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557679    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy"
Oct 31 19:41:29.557989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557766    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy"
Oct 31 19:41:29.557989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557774    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4429158d-d586-4a08-81bf-ba74734f82fc" containerName="maas-api"
Oct 31 19:41:29.557989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557784    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32f2799-c8ee-418d-ba27-31d90cba3a40" containerName="istio-proxy"
Oct 31 19:41:29.557989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557791    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy"
Oct 31 19:41:29.557989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557796    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1507e13-825a-46b7-97bd-dbf780d549c3" containerName="istio-proxy"
Oct 31 19:41:29.557989 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.557801    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy"
Oct 31 19:41:29.561057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.561040    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.612859 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.612833    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e4a2c-openshift-default-dockercfg-7jr6n\""
Oct 31 19:41:29.639418 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639385    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497282da-94c7-49aa-a477-9185d0603804-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.639569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639483    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.639617 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639568    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.639654 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639623    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.639712 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639697    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.639782 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639769    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.639816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639809    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497282da-94c7-49aa-a477-9185d0603804-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.639848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639829    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jkq4\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-kube-api-access-2jkq4\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.639883 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.639846    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.665603 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.665575    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"]
Oct 31 19:41:29.741325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741290    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497282da-94c7-49aa-a477-9185d0603804-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741392    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741442    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741513    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741734 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741579    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741734 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741660    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741817    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497282da-94c7-49aa-a477-9185d0603804-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741853    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741869    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jkq4\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-kube-api-access-2jkq4\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.741907 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741890    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.742071 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.741905    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.742071 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.742056    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.742150 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.742100    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497282da-94c7-49aa-a477-9185d0603804-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.742200 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.742181    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.744055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.744026    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.744305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.744286    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497282da-94c7-49aa-a477-9185d0603804-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.804901 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.804869    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jkq4\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-kube-api-access-2jkq4\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.805864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.805841    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:29.875178 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:29.875149    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:30.312579 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:30.312555    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"]
Oct 31 19:41:30.314228 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:41:30.314200    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497282da_94c7_49aa_a477_9185d0603804.slice/crio-773ef5061c4017dc761e2a773aabd41e66be821f8383e72c9f399fe99bffdfcc WatchSource:0}: Error finding container 773ef5061c4017dc761e2a773aabd41e66be821f8383e72c9f399fe99bffdfcc: Status 404 returned error can't find the container with id 773ef5061c4017dc761e2a773aabd41e66be821f8383e72c9f399fe99bffdfcc
Oct 31 19:41:30.316756 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:30.316724    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:41:30.316847 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:30.316786    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:41:30.316847 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:30.316814    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:41:30.937080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:30.937042    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" event={"ID":"497282da-94c7-49aa-a477-9185d0603804","Type":"ContainerStarted","Data":"6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430"}
Oct 31 19:41:30.937510 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:30.937102    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" event={"ID":"497282da-94c7-49aa-a477-9185d0603804","Type":"ContainerStarted","Data":"773ef5061c4017dc761e2a773aabd41e66be821f8383e72c9f399fe99bffdfcc"}
Oct 31 19:41:31.122007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:31.121955    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" podStartSLOduration=2.121938779 podStartE2EDuration="2.121938779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:31.119212061 +0000 UTC m=+4794.350450354" watchObservedRunningTime="2026-10-31 19:41:31.121938779 +0000 UTC m=+4794.353177068"
Oct 31 19:41:31.876589 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:31.876559    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:31.877247 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:31.877223    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:31.877358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:31.877279    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 19:41:32.876079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:32.876046    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:32.876661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:32.876626    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 19:41:33.876016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:33.875971    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:33.876215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:33.876043    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 19:41:34.881874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:34.881832    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:34.882483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:34.882429    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 19:41:35.875982 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:35.875945    5551 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:35.876176 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:35.876007    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 19:41:36.881266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:36.881217    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:36.882377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:36.882358    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:36.883159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:36.883123    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:37.153596 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:37.153510    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"]
Oct 31 19:41:38.010711 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:38.010679    5551 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" secret="" err="secret \"e2e-worker-w3-3e4a2c-openshift-default-dockercfg-7jr6n\" not found"
Oct 31 19:41:39.018032 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:39.017983    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy" containerID="cri-o://6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430" gracePeriod=30
Oct 31 19:41:40.665433 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:40.665404    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:41:40.667508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:40.667485    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:41:40.709270 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:40.709240    5551 scope.go:122] "RemoveContainer" containerID="4c409f2e39942ff013fcf709116aaa58f280c3dc6efdffce99c267739ae95ba5"
Oct 31 19:41:44.309162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.309135    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:44.437107 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437065    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-envoy\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437133    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-kube-api-access-2jkq4\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-kube-api-access-2jkq4\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437180    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-certs\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437359 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437332    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-istio-token\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-istio-token\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437417 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437380    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-socket\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437423    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/497282da-94c7-49aa-a477-9185d0603804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497282da-94c7-49aa-a477-9185d0603804-istio-podinfo\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437496    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-data\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437519    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497282da-94c7-49aa-a477-9185d0603804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497282da-94c7-49aa-a477-9185d0603804-istiod-ca-cert\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437528    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-certs" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:44.437723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437567    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-credential-socket\") pod \"497282da-94c7-49aa-a477-9185d0603804\" (UID: \"497282da-94c7-49aa-a477-9185d0603804\") "
Oct 31 19:41:44.437723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437619    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-socket" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:44.437846 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437741    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-data" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:44.437969 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.437947    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-credential-socket" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:44.438210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.438136    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.438210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.438162    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497282da-94c7-49aa-a477-9185d0603804-istiod-ca-cert" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:44.438210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.438162    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.438403 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.438268    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.438403 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.438352    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.439792 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.439763    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/497282da-94c7-49aa-a477-9185d0603804-istio-podinfo" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:44.439879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.439830    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-istio-token" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:44.440089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.440072    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-envoy" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:44.440141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.440081    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-kube-api-access-2jkq4" pod "497282da-94c7-49aa-a477-9185d0603804" (UID: "497282da-94c7-49aa-a477-9185d0603804"). InnerVolumeSpecName "kube-api-access-2jkq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:44.539080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.538999    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497282da-94c7-49aa-a477-9185d0603804-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.539080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.539030    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jkq4\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-kube-api-access-2jkq4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.539080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.539041    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497282da-94c7-49aa-a477-9185d0603804-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.539080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.539049    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497282da-94c7-49aa-a477-9185d0603804-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.539080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:44.539058    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497282da-94c7-49aa-a477-9185d0603804-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:45.061102 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.061069    5551 generic.go:356] "Generic (PLEG): container finished" podID="497282da-94c7-49aa-a477-9185d0603804" containerID="6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430" exitCode=0
Oct 31 19:41:45.061308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.061133    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"
Oct 31 19:41:45.061308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.061161    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" event={"ID":"497282da-94c7-49aa-a477-9185d0603804","Type":"ContainerDied","Data":"6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430"}
Oct 31 19:41:45.061308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.061227    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr" event={"ID":"497282da-94c7-49aa-a477-9185d0603804","Type":"ContainerDied","Data":"773ef5061c4017dc761e2a773aabd41e66be821f8383e72c9f399fe99bffdfcc"}
Oct 31 19:41:45.061308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.061258    5551 scope.go:122] "RemoveContainer" containerID="6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430"
Oct 31 19:41:45.075349 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.075330    5551 scope.go:122] "RemoveContainer" containerID="6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430"
Oct 31 19:41:45.075709 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:41:45.075685    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430\": container with ID starting with 6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430 not found: ID does not exist" containerID="6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430"
Oct 31 19:41:45.075808 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.075723    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430"} err="failed to get container status \"6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430\": rpc error: code = NotFound desc = could not find container \"6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430\": container with ID starting with 6bd78b68fc3d2cacd67a498a8b8f3fb54a1aa34092d2d6d02875ac19a59a9430 not found: ID does not exist"
Oct 31 19:41:45.277088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.277007    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"]
Oct 31 19:41:45.323377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:45.323346    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-kxxnr"]
Oct 31 19:41:46.709480 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:41:46.708694    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:41:46.911508 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:41:46.911452    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497282da-94c7-49aa-a477-9185d0603804" path="/var/lib/kubelet/pods/497282da-94c7-49aa-a477-9185d0603804/volumes"
Oct 31 19:42:01.121278 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:01.121238    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"]
Oct 31 19:42:01.121879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:01.121552    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy" containerID="cri-o://8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12" gracePeriod=30
Oct 31 19:42:06.383447 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.383420    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:42:06.406636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406566    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-token\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-token\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.406780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406639    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-certs\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.406780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406690    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-kube-api-access-59bjb\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-kube-api-access-59bjb\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.406780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406724    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b869d2ac-8f69-4e85-8669-814ea8927d62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b869d2ac-8f69-4e85-8669-814ea8927d62-istiod-ca-cert\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.406780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406773    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-data\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.407005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406811    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-credential-socket\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.407005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406859    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-podinfo\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.407005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406916    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-socket\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.407005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406963    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-envoy\") pod \"b869d2ac-8f69-4e85-8669-814ea8927d62\" (UID: \"b869d2ac-8f69-4e85-8669-814ea8927d62\") "
Oct 31 19:42:06.407232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.406986    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-certs" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:06.407232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407055    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-data" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:06.407232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407128    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-credential-socket" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:06.407232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407193    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b869d2ac-8f69-4e85-8669-814ea8927d62-istiod-ca-cert" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:06.407439 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407284    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-socket" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:06.407674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407653    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:06.407758 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407677    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b869d2ac-8f69-4e85-8669-814ea8927d62-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:06.407758 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407691    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:06.407758 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407704    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:06.407758 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.407718    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:06.409076 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.409051    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-token" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:06.409246 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.409232    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-podinfo" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:42:06.409629 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.409608    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-kube-api-access-59bjb" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "kube-api-access-59bjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:06.409721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.409704    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-envoy" pod "b869d2ac-8f69-4e85-8669-814ea8927d62" (UID: "b869d2ac-8f69-4e85-8669-814ea8927d62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:06.508938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.508904    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:06.508938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.508932    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:06.508938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.508941    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:06.509163 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:06.508950    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59bjb\" (UniqueName: \"kubernetes.io/projected/b869d2ac-8f69-4e85-8669-814ea8927d62-kube-api-access-59bjb\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:07.211501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.211371    5551 generic.go:356] "Generic (PLEG): container finished" podID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerID="8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12" exitCode=0
Oct 31 19:42:07.211501 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.211486    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"
Oct 31 19:42:07.211710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.211481    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" event={"ID":"b869d2ac-8f69-4e85-8669-814ea8927d62","Type":"ContainerDied","Data":"8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12"}
Oct 31 19:42:07.211710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.211615    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9" event={"ID":"b869d2ac-8f69-4e85-8669-814ea8927d62","Type":"ContainerDied","Data":"ed5fe74903dfc48601f268b42c01d4fa220486187f587ee16ee334d228cbe78f"}
Oct 31 19:42:07.211710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.211637    5551 scope.go:122] "RemoveContainer" containerID="8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12"
Oct 31 19:42:07.222021 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.222001    5551 scope.go:122] "RemoveContainer" containerID="3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5"
Oct 31 19:42:07.231162 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.231142    5551 scope.go:122] "RemoveContainer" containerID="8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12"
Oct 31 19:42:07.231434 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:42:07.231412    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12\": container with ID starting with 8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12 not found: ID does not exist" containerID="8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12"
Oct 31 19:42:07.231560 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.231441    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12"} err="failed to get container status \"8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12\": rpc error: code = NotFound desc = could not find container \"8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12\": container with ID starting with 8a6ae0c6c272a11f62a2fb91ddb3ebffff7de6486160603d5a5cfc1d2c3fac12 not found: ID does not exist"
Oct 31 19:42:07.231609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.231564    5551 scope.go:122] "RemoveContainer" containerID="3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5"
Oct 31 19:42:07.231830 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:42:07.231814    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5\": container with ID starting with 3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5 not found: ID does not exist" containerID="3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5"
Oct 31 19:42:07.231900 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.231834    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5"} err="failed to get container status \"3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5\": rpc error: code = NotFound desc = could not find container \"3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5\": container with ID starting with 3fac2a64665254b2538242db71e76ce3ec17c1a1e2ab078504d1cb49df3899a5 not found: ID does not exist"
Oct 31 19:42:07.377220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.377188    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"]
Oct 31 19:42:07.424656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:07.424630    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-xd9v9"]
Oct 31 19:42:08.910830 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:08.910793    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" path="/var/lib/kubelet/pods/b869d2ac-8f69-4e85-8669-814ea8927d62/volumes"
Oct 31 19:42:16.605159 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:42:16.605119    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:42:19.349826 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:19.349788    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"]
Oct 31 19:42:24.350803 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:24.350754    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr" podUID="46532445-f525-4dfd-ac76-c0d2e7c26f2d" containerName="payload-pre-processing" containerID="cri-o://51700b711015c9d9e83c5b2fb3fcb0413538996367e448df69723ec745e26cbb" gracePeriod=25
Oct 31 19:42:28.253150 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:28.253100    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr" podUID="46532445-f525-4dfd-ac76-c0d2e7c26f2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:42:33.337142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:33.337105    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"]
Oct 31 19:42:38.253578 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:38.253539    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr" podUID="46532445-f525-4dfd-ac76-c0d2e7c26f2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:42:44.097070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.096974    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"]
Oct 31 19:42:44.097759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.097737    5551 cpu_manager.go:414] "containerMap: removing container" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy"
Oct 31 19:42:44.097829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.097763    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebec53a-ceec-4521-8d79-b345aeefef4f" containerName="istio-proxy"
Oct 31 19:42:44.097829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.097780    5551 cpu_manager.go:414] "containerMap: removing container" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy"
Oct 31 19:42:44.097829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.097785    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy"
Oct 31 19:42:44.097829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.097796    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy"
Oct 31 19:42:44.097829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.097802    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy"
Oct 31 19:42:44.097996 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.097917    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy"
Oct 31 19:42:44.097996 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.097929    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="497282da-94c7-49aa-a477-9185d0603804" containerName="istio-proxy"
Oct 31 19:42:44.102929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.102899    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.161110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.159536    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6279f5-openshift-default-dockercfg-d8cq6\""
Oct 31 19:42:44.183648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.183596    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.183955 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.183902    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.184138 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.184112    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.184272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.184153    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.184272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.184215    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.184392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.184277    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.184392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.184307    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjs98\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-kube-api-access-pjs98\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.184392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.184338    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.184652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.184441    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.213506 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.213428    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"]
Oct 31 19:42:44.286300 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286258    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.286545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286325    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.286545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286345    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjs98\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-kube-api-access-pjs98\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.286545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286366    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.286545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286386    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.286545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286497    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.286545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286542    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.287311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286621    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.287311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286652    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.287311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286870    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.287311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286911    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.287311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286936    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.287311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.286984    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.287576 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.287315    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.288856 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.288838    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.289077 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.289054    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.345096 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.345054    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.345253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.345127    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjs98\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-kube-api-access-pjs98\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.419053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.419011    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:44.478621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.478586    5551 generic.go:356] "Generic (PLEG): container finished" podID="46532445-f525-4dfd-ac76-c0d2e7c26f2d" containerID="51700b711015c9d9e83c5b2fb3fcb0413538996367e448df69723ec745e26cbb" exitCode=0
Oct 31 19:42:44.478756 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.478734    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr" event={"ID":"46532445-f525-4dfd-ac76-c0d2e7c26f2d","Type":"ContainerDied","Data":"51700b711015c9d9e83c5b2fb3fcb0413538996367e448df69723ec745e26cbb"}
Oct 31 19:42:44.657596 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.657568    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 19:42:44.792974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.792939    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46532445-f525-4dfd-ac76-c0d2e7c26f2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46532445-f525-4dfd-ac76-c0d2e7c26f2d-plugins-config-volume\") pod \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\" (UID: \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\") "
Oct 31 19:42:44.793169 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.792979    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46532445-f525-4dfd-ac76-c0d2e7c26f2d-kube-api-access-jwqbn\" (UniqueName: \"kubernetes.io/projected/46532445-f525-4dfd-ac76-c0d2e7c26f2d-kube-api-access-jwqbn\") pod \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\" (UID: \"46532445-f525-4dfd-ac76-c0d2e7c26f2d\") "
Oct 31 19:42:44.793376 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.793352    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46532445-f525-4dfd-ac76-c0d2e7c26f2d-plugins-config-volume" pod "46532445-f525-4dfd-ac76-c0d2e7c26f2d" (UID: "46532445-f525-4dfd-ac76-c0d2e7c26f2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:44.795285 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.795259    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46532445-f525-4dfd-ac76-c0d2e7c26f2d-kube-api-access-jwqbn" pod "46532445-f525-4dfd-ac76-c0d2e7c26f2d" (UID: "46532445-f525-4dfd-ac76-c0d2e7c26f2d"). InnerVolumeSpecName "kube-api-access-jwqbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:44.854726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.854697    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"]
Oct 31 19:42:44.856381 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:42:44.856355    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7ec9b1_6584_4840_a1b7_8b4ae5f2a85a.slice/crio-886d9ce3275abf021ae7a5b3018d9a1276b50b993fe4f9ee9a48923619ff44f9 WatchSource:0}: Error finding container 886d9ce3275abf021ae7a5b3018d9a1276b50b993fe4f9ee9a48923619ff44f9: Status 404 returned error can't find the container with id 886d9ce3275abf021ae7a5b3018d9a1276b50b993fe4f9ee9a48923619ff44f9
Oct 31 19:42:44.858710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.858677    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:42:44.858807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.858739    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:42:44.858807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.858768    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:42:44.893958 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.893930    5551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46532445-f525-4dfd-ac76-c0d2e7c26f2d-plugins-config-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:44.893958 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:44.893956    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwqbn\" (UniqueName: \"kubernetes.io/projected/46532445-f525-4dfd-ac76-c0d2e7c26f2d-kube-api-access-jwqbn\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:45.494073 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:45.494023    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" event={"ID":"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a","Type":"ContainerStarted","Data":"0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9"}
Oct 31 19:42:45.494601 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:45.494137    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" event={"ID":"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a","Type":"ContainerStarted","Data":"886d9ce3275abf021ae7a5b3018d9a1276b50b993fe4f9ee9a48923619ff44f9"}
Oct 31 19:42:45.495708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:45.495685    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"
Oct 31 19:42:45.495814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:45.495708    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr" event={"ID":"46532445-f525-4dfd-ac76-c0d2e7c26f2d","Type":"ContainerDied","Data":"845738a272e14ee700cb6ffc8e0cc47b45f0402638ec171056f5bfdcddc16fe3"}
Oct 31 19:42:45.495814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:45.495763    5551 scope.go:122] "RemoveContainer" containerID="51700b711015c9d9e83c5b2fb3fcb0413538996367e448df69723ec745e26cbb"
Oct 31 19:42:45.625088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:45.625038    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podStartSLOduration=1.6250189160000001 podStartE2EDuration="1.625018916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:45.620661475 +0000 UTC m=+4868.851899789" watchObservedRunningTime="2026-10-31 19:42:45.625018916 +0000 UTC m=+4868.856257207"
Oct 31 19:42:45.734908 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:45.734873    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"]
Oct 31 19:42:45.739931 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:45.739900    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6279f5-7746654c8c-gnwjr"]
Oct 31 19:42:46.419421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:46.419387    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:46.421130 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:46.421103    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 19:42:46.421273 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:46.421159    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 19:42:46.609013 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:42:46.608971    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:42:46.908866 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:46.908834    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46532445-f525-4dfd-ac76-c0d2e7c26f2d" path="/var/lib/kubelet/pods/46532445-f525-4dfd-ac76-c0d2e7c26f2d/volumes"
Oct 31 19:42:47.419633 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:47.419604    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 19:42:47.419815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:47.419665    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 19:42:48.359796 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.359752    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f" podUID="88b55310-85c9-403e-98de-4be997a7aeaf" containerName="main" containerID="cri-o://38feea745597edf3a3e509f05cadadb8a5c443d068e0e220f03f754321454886" gracePeriod=45
Oct 31 19:42:48.419807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.419776    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 19:42:48.419928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.419841    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 19:42:48.517559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.517524    5551 generic.go:356] "Generic (PLEG): container finished" podID="88b55310-85c9-403e-98de-4be997a7aeaf" containerID="38feea745597edf3a3e509f05cadadb8a5c443d068e0e220f03f754321454886" exitCode=0
Oct 31 19:42:48.517701 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.517585    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f" event={"ID":"88b55310-85c9-403e-98de-4be997a7aeaf","Type":"ContainerDied","Data":"38feea745597edf3a3e509f05cadadb8a5c443d068e0e220f03f754321454886"}
Oct 31 19:42:48.688493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.688447    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:42:48.739422 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739391    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-dshm\") pod \"88b55310-85c9-403e-98de-4be997a7aeaf\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") "
Oct 31 19:42:48.739617 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739559    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-tmp-dir\") pod \"88b55310-85c9-403e-98de-4be997a7aeaf\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") "
Oct 31 19:42:48.739617 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739604    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b55310-85c9-403e-98de-4be997a7aeaf-kube-api-access-qmmrl\" (UniqueName: \"kubernetes.io/projected/88b55310-85c9-403e-98de-4be997a7aeaf-kube-api-access-qmmrl\") pod \"88b55310-85c9-403e-98de-4be997a7aeaf\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") "
Oct 31 19:42:48.739754 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739657    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-home\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-home\") pod \"88b55310-85c9-403e-98de-4be997a7aeaf\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") "
Oct 31 19:42:48.739754 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739716    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-model-cache\") pod \"88b55310-85c9-403e-98de-4be997a7aeaf\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") "
Oct 31 19:42:48.739754 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739749    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b55310-85c9-403e-98de-4be997a7aeaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/88b55310-85c9-403e-98de-4be997a7aeaf-tls-certs\") pod \"88b55310-85c9-403e-98de-4be997a7aeaf\" (UID: \"88b55310-85c9-403e-98de-4be997a7aeaf\") "
Oct 31 19:42:48.739924 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739807    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-tmp-dir" pod "88b55310-85c9-403e-98de-4be997a7aeaf" (UID: "88b55310-85c9-403e-98de-4be997a7aeaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:48.739994 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739941    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-home" pod "88b55310-85c9-403e-98de-4be997a7aeaf" (UID: "88b55310-85c9-403e-98de-4be997a7aeaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:48.739994 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.739975    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-model-cache" pod "88b55310-85c9-403e-98de-4be997a7aeaf" (UID: "88b55310-85c9-403e-98de-4be997a7aeaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:48.740266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.740210    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:48.740266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.740235    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:48.740266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.740245    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:48.741960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.741934    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-dshm" pod "88b55310-85c9-403e-98de-4be997a7aeaf" (UID: "88b55310-85c9-403e-98de-4be997a7aeaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:48.742479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.742441    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b55310-85c9-403e-98de-4be997a7aeaf-tls-certs" pod "88b55310-85c9-403e-98de-4be997a7aeaf" (UID: "88b55310-85c9-403e-98de-4be997a7aeaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:48.742570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.742487    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b55310-85c9-403e-98de-4be997a7aeaf-kube-api-access-qmmrl" pod "88b55310-85c9-403e-98de-4be997a7aeaf" (UID: "88b55310-85c9-403e-98de-4be997a7aeaf"). InnerVolumeSpecName "kube-api-access-qmmrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:48.841392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.841355    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b55310-85c9-403e-98de-4be997a7aeaf-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:48.841392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.841385    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b55310-85c9-403e-98de-4be997a7aeaf-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:48.841392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:48.841401    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmrl\" (UniqueName: \"kubernetes.io/projected/88b55310-85c9-403e-98de-4be997a7aeaf-kube-api-access-qmmrl\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:49.266088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.264083    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"]
Oct 31 19:42:49.266088 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.264329    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w" podUID="b1993cb5-c116-4c47-be36-3893da5088c9" containerName="maas-api" containerID="cri-o://8c6647a3d5707656c44d60af8621520b833ec88d533324b696522cb44a9b402e" gracePeriod=30
Oct 31 19:42:49.419825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.419791    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 19:42:49.420210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.419856    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 19:42:49.529521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.529298    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f" event={"ID":"88b55310-85c9-403e-98de-4be997a7aeaf","Type":"ContainerDied","Data":"067d14b6c7413ead511bfae2f825eb8f05f606f015244bd3260f3b1549d3d1e9"}
Oct 31 19:42:49.529521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.529359    5551 scope.go:122] "RemoveContainer" containerID="38feea745597edf3a3e509f05cadadb8a5c443d068e0e220f03f754321454886"
Oct 31 19:42:49.529521 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.529372    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"
Oct 31 19:42:49.532556 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.532531    5551 generic.go:356] "Generic (PLEG): container finished" podID="b1993cb5-c116-4c47-be36-3893da5088c9" containerID="8c6647a3d5707656c44d60af8621520b833ec88d533324b696522cb44a9b402e" exitCode=0
Oct 31 19:42:49.532675 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.532599    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w" event={"ID":"b1993cb5-c116-4c47-be36-3893da5088c9","Type":"ContainerDied","Data":"8c6647a3d5707656c44d60af8621520b833ec88d533324b696522cb44a9b402e"}
Oct 31 19:42:49.622290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.622259    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 19:42:49.713817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.713779    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"]
Oct 31 19:42:49.716631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.716606    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-dbceb9/sub-test-model-dbceb9-kserve-68cd68ddff-bw85f"]
Oct 31 19:42:49.756482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.753833    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls\") pod \"b1993cb5-c116-4c47-be36-3893da5088c9\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") "
Oct 31 19:42:49.756482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.753944    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1993cb5-c116-4c47-be36-3893da5088c9-kube-api-access-dpvlt\" (UniqueName: \"kubernetes.io/projected/b1993cb5-c116-4c47-be36-3893da5088c9-kube-api-access-dpvlt\") pod \"b1993cb5-c116-4c47-be36-3893da5088c9\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") "
Oct 31 19:42:49.756482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.754024    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-metrics-certs\") pod \"b1993cb5-c116-4c47-be36-3893da5088c9\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") "
Oct 31 19:42:49.756482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.754117    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1993cb5-c116-4c47-be36-3893da5088c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1993cb5-c116-4c47-be36-3893da5088c9-openshift-service-ca\") pod \"b1993cb5-c116-4c47-be36-3893da5088c9\" (UID: \"b1993cb5-c116-4c47-be36-3893da5088c9\") "
Oct 31 19:42:49.756482 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.754857    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1993cb5-c116-4c47-be36-3893da5088c9-openshift-service-ca" pod "b1993cb5-c116-4c47-be36-3893da5088c9" (UID: "b1993cb5-c116-4c47-be36-3893da5088c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:49.775483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.768080    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-metrics-certs" pod "b1993cb5-c116-4c47-be36-3893da5088c9" (UID: "b1993cb5-c116-4c47-be36-3893da5088c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:49.775483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.768472    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls" pod "b1993cb5-c116-4c47-be36-3893da5088c9" (UID: "b1993cb5-c116-4c47-be36-3893da5088c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:49.783538 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.781880    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1993cb5-c116-4c47-be36-3893da5088c9-kube-api-access-dpvlt" pod "b1993cb5-c116-4c47-be36-3893da5088c9" (UID: "b1993cb5-c116-4c47-be36-3893da5088c9"). InnerVolumeSpecName "kube-api-access-dpvlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:49.889504 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.886560    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvlt\" (UniqueName: \"kubernetes.io/projected/b1993cb5-c116-4c47-be36-3893da5088c9-kube-api-access-dpvlt\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:49.889504 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.886596    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:49.889504 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.886614    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1993cb5-c116-4c47-be36-3893da5088c9-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:49.889504 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:49.886638    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1993cb5-c116-4c47-be36-3893da5088c9-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:50.419421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.419383    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 19:42:50.419628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.419437    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 19:42:50.541806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.541772    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w" event={"ID":"b1993cb5-c116-4c47-be36-3893da5088c9","Type":"ContainerDied","Data":"b4570defbfbaf2df8470b0ad674f0b530ff6fa05125446aa2af6bc94043cba83"}
Oct 31 19:42:50.541806 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.541807    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"
Oct 31 19:42:50.542314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.541836    5551 scope.go:122] "RemoveContainer" containerID="8c6647a3d5707656c44d60af8621520b833ec88d533324b696522cb44a9b402e"
Oct 31 19:42:50.704736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.704648    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"]
Oct 31 19:42:50.768483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.758142    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6279f5-6f85ff9b8f-kh89w"]
Oct 31 19:42:50.909235 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.909197    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b55310-85c9-403e-98de-4be997a7aeaf" path="/var/lib/kubelet/pods/88b55310-85c9-403e-98de-4be997a7aeaf/volumes"
Oct 31 19:42:50.909740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:50.909722    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1993cb5-c116-4c47-be36-3893da5088c9" path="/var/lib/kubelet/pods/b1993cb5-c116-4c47-be36-3893da5088c9/volumes"
Oct 31 19:42:51.419653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:51.419615    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 19:42:51.419862 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:51.419685    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 19:42:53.460591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:53.460559    5551 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:53.461074 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:53.460638    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:53.676312 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:53.676274    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:53.676871 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:53.676809    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:53.677898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:53.677881    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:42:56.619985 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:56.619950    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"]
Oct 31 19:42:59.335321 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:59.335285    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"]
Oct 31 19:42:59.335934 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:42:59.335493    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy" containerID="cri-o://0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9" gracePeriod=30
Oct 31 19:43:04.602927 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.602846    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:43:04.679034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.679004    5551 generic.go:356] "Generic (PLEG): container finished" podID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerID="0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9" exitCode=0
Oct 31 19:43:04.679233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.679127    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"
Oct 31 19:43:04.679233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.679152    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" event={"ID":"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a","Type":"ContainerDied","Data":"0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9"}
Oct 31 19:43:04.679233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.679202    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j" event={"ID":"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a","Type":"ContainerDied","Data":"886d9ce3275abf021ae7a5b3018d9a1276b50b993fe4f9ee9a48923619ff44f9"}
Oct 31 19:43:04.679233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.679224    5551 scope.go:122] "RemoveContainer" containerID="0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9"
Oct 31 19:43:04.684502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684478    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-token\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.684630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684573    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istiod-ca-cert\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.684630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684598    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-certs\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.684757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684665    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-envoy\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.684757 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684726    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-credential-socket\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.684865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684779    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-kube-api-access-pjs98\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-kube-api-access-pjs98\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.684865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684819    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-podinfo\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.684865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684861    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-data\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.685040 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684955    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-certs" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.685040 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.684969    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-socket\") pod \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\" (UID: \"5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a\") "
Oct 31 19:43:04.685159 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685044    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-credential-socket" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.685389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685191    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-data" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.685389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685282    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-socket" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.685389 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685354    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istiod-ca-cert" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:04.685598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685503    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.685598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685524    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.685598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685539    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.685598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685553    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.685598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.685568    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.686918 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.686892    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-token" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:04.687207 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.687178    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-kube-api-access-pjs98" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "kube-api-access-pjs98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:04.687360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.687335    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-podinfo" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:43:04.687570 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.687548    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-envoy" pod "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" (UID: "5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.693648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.693630    5551 scope.go:122] "RemoveContainer" containerID="0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9"
Oct 31 19:43:04.694008 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:43:04.693988    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9\": container with ID starting with 0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9 not found: ID does not exist" containerID="0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9"
Oct 31 19:43:04.694067 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.694018    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9"} err="failed to get container status \"0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9\": rpc error: code = NotFound desc = could not find container \"0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9\": container with ID starting with 0457351a9f219d60bd12ca89e7e70fc3ee09f772f4e394af0390f0c162506bf9 not found: ID does not exist"
Oct 31 19:43:04.786958 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.786922    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjs98\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-kube-api-access-pjs98\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.786958 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.786958    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.787210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.786970    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.787210 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:04.786980    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:05.098865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:05.098835    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"]
Oct 31 19:43:05.108141 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:05.108111    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-g7h5j"]
Oct 31 19:43:06.909831 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:06.909795    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" path="/var/lib/kubelet/pods/5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a/volumes"
Oct 31 19:43:11.641084 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.641035    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn" podUID="abfb49ee-7672-4e09-be02-f0dc4aa237de" containerName="main" containerID="cri-o://3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b" gracePeriod=45
Oct 31 19:43:11.898789 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.898726    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:43:11.974276 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974235    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-home\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-home\") pod \"abfb49ee-7672-4e09-be02-f0dc4aa237de\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") "
Oct 31 19:43:11.974511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974341    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-dshm\") pod \"abfb49ee-7672-4e09-be02-f0dc4aa237de\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") "
Oct 31 19:43:11.974511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974425    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfb49ee-7672-4e09-be02-f0dc4aa237de-kube-api-access-lw8j4\" (UniqueName: \"kubernetes.io/projected/abfb49ee-7672-4e09-be02-f0dc4aa237de-kube-api-access-lw8j4\") pod \"abfb49ee-7672-4e09-be02-f0dc4aa237de\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") "
Oct 31 19:43:11.974511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974449    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-tmp-dir\") pod \"abfb49ee-7672-4e09-be02-f0dc4aa237de\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") "
Oct 31 19:43:11.974511 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974506    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abfb49ee-7672-4e09-be02-f0dc4aa237de-tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb49ee-7672-4e09-be02-f0dc4aa237de-tls-certs\") pod \"abfb49ee-7672-4e09-be02-f0dc4aa237de\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") "
Oct 31 19:43:11.974780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974513    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-home" pod "abfb49ee-7672-4e09-be02-f0dc4aa237de" (UID: "abfb49ee-7672-4e09-be02-f0dc4aa237de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:11.974780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974667    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-model-cache\") pod \"abfb49ee-7672-4e09-be02-f0dc4aa237de\" (UID: \"abfb49ee-7672-4e09-be02-f0dc4aa237de\") "
Oct 31 19:43:11.974780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974760    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-tmp-dir" pod "abfb49ee-7672-4e09-be02-f0dc4aa237de" (UID: "abfb49ee-7672-4e09-be02-f0dc4aa237de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:11.974949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.974907    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-model-cache" pod "abfb49ee-7672-4e09-be02-f0dc4aa237de" (UID: "abfb49ee-7672-4e09-be02-f0dc4aa237de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:11.975128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.975111    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:11.975364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.975129    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:11.975364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.975143    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:11.976838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.976807    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfb49ee-7672-4e09-be02-f0dc4aa237de-kube-api-access-lw8j4" pod "abfb49ee-7672-4e09-be02-f0dc4aa237de" (UID: "abfb49ee-7672-4e09-be02-f0dc4aa237de"). InnerVolumeSpecName "kube-api-access-lw8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:11.976938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.976856    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfb49ee-7672-4e09-be02-f0dc4aa237de-tls-certs" pod "abfb49ee-7672-4e09-be02-f0dc4aa237de" (UID: "abfb49ee-7672-4e09-be02-f0dc4aa237de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:11.976938 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:11.976887    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-dshm" pod "abfb49ee-7672-4e09-be02-f0dc4aa237de" (UID: "abfb49ee-7672-4e09-be02-f0dc4aa237de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:12.076423 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.076392    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abfb49ee-7672-4e09-be02-f0dc4aa237de-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:12.076423 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.076415    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abfb49ee-7672-4e09-be02-f0dc4aa237de-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:12.076423 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.076425    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8j4\" (UniqueName: \"kubernetes.io/projected/abfb49ee-7672-4e09-be02-f0dc4aa237de-kube-api-access-lw8j4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:12.732285 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.732255    5551 generic.go:356] "Generic (PLEG): container finished" podID="abfb49ee-7672-4e09-be02-f0dc4aa237de" containerID="3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b" exitCode=0
Oct 31 19:43:12.732285 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.732290    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn" event={"ID":"abfb49ee-7672-4e09-be02-f0dc4aa237de","Type":"ContainerDied","Data":"3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b"}
Oct 31 19:43:12.732879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.732303    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"
Oct 31 19:43:12.732879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.732315    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn" event={"ID":"abfb49ee-7672-4e09-be02-f0dc4aa237de","Type":"ContainerDied","Data":"a36fff910908389565329bf822c5370413f878bc072f8e3d8f9f2aa683d55cf9"}
Oct 31 19:43:12.732879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.732332    5551 scope.go:122] "RemoveContainer" containerID="3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b"
Oct 31 19:43:12.742935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.742918    5551 scope.go:122] "RemoveContainer" containerID="3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b"
Oct 31 19:43:12.743214 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:43:12.743191    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b\": container with ID starting with 3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b not found: ID does not exist" containerID="3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b"
Oct 31 19:43:12.743296 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.743225    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b"} err="failed to get container status \"3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b\": rpc error: code = NotFound desc = could not find container \"3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b\": container with ID starting with 3982fa733ed0f93b1fb3f9e3d4a8b91f700893708db49a8f783ac311f498c33b not found: ID does not exist"
Oct 31 19:43:12.898001 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.897971    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"]
Oct 31 19:43:12.901316 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.901291    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6cfd68/sub-test-model-6cfd68-kserve-7d8675bd55-q8nnn"]
Oct 31 19:43:12.909308 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:12.909277    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfb49ee-7672-4e09-be02-f0dc4aa237de" path="/var/lib/kubelet/pods/abfb49ee-7672-4e09-be02-f0dc4aa237de/volumes"
Oct 31 19:43:16.604442 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:43:16.604404    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:43:18.147915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:18.147878    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"]
Oct 31 19:43:18.148452 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:18.148101    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy" containerID="cri-o://4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa" gracePeriod=30
Oct 31 19:43:20.259889 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:20.259850    5551 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:43:20.260515 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:20.259930    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:43:23.406279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.406253    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:43:23.504241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504177    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-kube-api-access-bczdj\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-kube-api-access-bczdj\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504217    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-data\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504238    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-envoy\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504299    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-credential-socket\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504327    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-certs\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504364    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-podinfo\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504503    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-token\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-token\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504607    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-socket\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504641    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e4dd78-4759-4bdb-9523-33ad3808763f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e4dd78-4759-4bdb-9523-33ad3808763f-istiod-ca-cert\") pod \"f5e4dd78-4759-4bdb-9523-33ad3808763f\" (UID: \"f5e4dd78-4759-4bdb-9523-33ad3808763f\") "
Oct 31 19:43:23.504767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504660    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-credential-socket" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:23.504944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504803    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-data" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:23.504944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504880    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-certs" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:23.504944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.504902    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-socket" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:23.505131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.505117    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e4dd78-4759-4bdb-9523-33ad3808763f-istiod-ca-cert" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:23.505371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.505348    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.505572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.505546    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e4dd78-4759-4bdb-9523-33ad3808763f-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.505684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.505575    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.505684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.505592    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.505684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.505607    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.506741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.506715    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-envoy" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:23.506978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.506945    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-podinfo" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:43:23.507071 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.507028    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-kube-api-access-bczdj" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "kube-api-access-bczdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:23.507071 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.507045    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-token" pod "f5e4dd78-4759-4bdb-9523-33ad3808763f" (UID: "f5e4dd78-4759-4bdb-9523-33ad3808763f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:23.607004 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.606972    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.607004 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.607004    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bczdj\" (UniqueName: \"kubernetes.io/projected/f5e4dd78-4759-4bdb-9523-33ad3808763f-kube-api-access-bczdj\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.607181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.607019    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.607181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.607027    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e4dd78-4759-4bdb-9523-33ad3808763f-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:23.801993 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.801897    5551 generic.go:356] "Generic (PLEG): container finished" podID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerID="4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa" exitCode=0
Oct 31 19:43:23.801993 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.801946    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"
Oct 31 19:43:23.801993 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.801953    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" event={"ID":"f5e4dd78-4759-4bdb-9523-33ad3808763f","Type":"ContainerDied","Data":"4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa"}
Oct 31 19:43:23.802288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.802034    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv" event={"ID":"f5e4dd78-4759-4bdb-9523-33ad3808763f","Type":"ContainerDied","Data":"5a996e6edc8085e43d8a81262eb0e5e81a8d176822252717b2bb05ad7e842855"}
Oct 31 19:43:23.802288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.802065    5551 scope.go:122] "RemoveContainer" containerID="4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa"
Oct 31 19:43:23.815288 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.815270    5551 scope.go:122] "RemoveContainer" containerID="28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28"
Oct 31 19:43:23.824035 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.824015    5551 scope.go:122] "RemoveContainer" containerID="4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa"
Oct 31 19:43:23.824293 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:43:23.824273    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa\": container with ID starting with 4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa not found: ID does not exist" containerID="4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa"
Oct 31 19:43:23.824346 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.824304    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa"} err="failed to get container status \"4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa\": rpc error: code = NotFound desc = could not find container \"4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa\": container with ID starting with 4fe9786d32203bb66a66d9cdbf0313cfc309d2971d0588e7bc671abe4739e2fa not found: ID does not exist"
Oct 31 19:43:23.824346 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.824325    5551 scope.go:122] "RemoveContainer" containerID="28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28"
Oct 31 19:43:23.824567 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:43:23.824552    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28\": container with ID starting with 28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28 not found: ID does not exist" containerID="28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28"
Oct 31 19:43:23.824629 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.824571    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28"} err="failed to get container status \"28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28\": rpc error: code = NotFound desc = could not find container \"28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28\": container with ID starting with 28b2d7767e7b86b0bcc26d27e6e70809e21dff61b0502973c48971ad6a1e7d28 not found: ID does not exist"
Oct 31 19:43:23.966134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.966103    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"]
Oct 31 19:43:23.968342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:23.968320    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dbceb9-openshift-default-64b67bff89-jhhsv"]
Oct 31 19:43:24.910507 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:24.910442    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" path="/var/lib/kubelet/pods/f5e4dd78-4759-4bdb-9523-33ad3808763f/volumes"
Oct 31 19:43:46.604768 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:43:46.604734    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:43:50.241152 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:50.241113    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"]
Oct 31 19:43:50.241718 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:50.241352    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy" containerID="cri-o://81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458" gracePeriod=30
Oct 31 19:43:55.505980 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.505924    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:43:55.555282 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555253    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7315b844-4c65-4097-9656-ed99e582ce4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7315b844-4c65-4097-9656-ed99e582ce4c-istiod-ca-cert\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555334    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-socket\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555382    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-data\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555413    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-kube-api-access-qdwfh\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555441    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-envoy\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555498    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-certs\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555525    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-istio-token\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-istio-token\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555592    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7315b844-4c65-4097-9656-ed99e582ce4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7315b844-4c65-4097-9656-ed99e582ce4c-istio-podinfo\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555677    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-credential-socket\") pod \"7315b844-4c65-4097-9656-ed99e582ce4c\" (UID: \"7315b844-4c65-4097-9656-ed99e582ce4c\") "
Oct 31 19:43:55.555967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555681    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-socket" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.555967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555679    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7315b844-4c65-4097-9656-ed99e582ce4c-istiod-ca-cert" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:55.555967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555705    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-data" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.555967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.555928    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-credential-socket" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.556191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.556065    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.556191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.556077    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7315b844-4c65-4097-9656-ed99e582ce4c-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.556191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.556087    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.556191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.556095    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.556542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.556512    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-certs" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.557978 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.557955    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7315b844-4c65-4097-9656-ed99e582ce4c-istio-podinfo" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:43:55.558618 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.558594    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-istio-token" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:55.558701 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.558676    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-kube-api-access-qdwfh" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "kube-api-access-qdwfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:55.558701 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.558684    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-envoy" pod "7315b844-4c65-4097-9656-ed99e582ce4c" (UID: "7315b844-4c65-4097-9656-ed99e582ce4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.657768 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.657726    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdwfh\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-kube-api-access-qdwfh\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.657768 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.657767    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.657973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.657782    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7315b844-4c65-4097-9656-ed99e582ce4c-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.657973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.657794    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7315b844-4c65-4097-9656-ed99e582ce4c-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.657973 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.657809    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7315b844-4c65-4097-9656-ed99e582ce4c-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.727197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:55.727164    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"]
Oct 31 19:43:56.004793 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.004759    5551 generic.go:356] "Generic (PLEG): container finished" podID="7315b844-4c65-4097-9656-ed99e582ce4c" containerID="81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458" exitCode=0
Oct 31 19:43:56.004975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.004806    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"
Oct 31 19:43:56.004975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.004844    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" event={"ID":"7315b844-4c65-4097-9656-ed99e582ce4c","Type":"ContainerDied","Data":"81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458"}
Oct 31 19:43:56.004975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.004899    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd" event={"ID":"7315b844-4c65-4097-9656-ed99e582ce4c","Type":"ContainerDied","Data":"985a85cd088636c79017dd0495f518e10465c94ef52723ada9269ae716b42c9e"}
Oct 31 19:43:56.004975 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.004924    5551 scope.go:122] "RemoveContainer" containerID="81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458"
Oct 31 19:43:56.015592 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.015575    5551 scope.go:122] "RemoveContainer" containerID="9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0"
Oct 31 19:43:56.024144 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.024127    5551 scope.go:122] "RemoveContainer" containerID="81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458"
Oct 31 19:43:56.024405 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:43:56.024389    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458\": container with ID starting with 81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458 not found: ID does not exist" containerID="81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458"
Oct 31 19:43:56.024445 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.024415    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458"} err="failed to get container status \"81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458\": rpc error: code = NotFound desc = could not find container \"81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458\": container with ID starting with 81e2a36aceb15905d0276c21e158e762c2e322d04f5bfb9f1d6c77f4b3297458 not found: ID does not exist"
Oct 31 19:43:56.024445 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.024431    5551 scope.go:122] "RemoveContainer" containerID="9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0"
Oct 31 19:43:56.024683 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:43:56.024668    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0\": container with ID starting with 9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0 not found: ID does not exist" containerID="9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0"
Oct 31 19:43:56.024721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.024689    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0"} err="failed to get container status \"9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0\": rpc error: code = NotFound desc = could not find container \"9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0\": container with ID starting with 9ca06910a980acea0d7b77fa5f3ac386d9c9695a364fe6130c73bc44784d2ed0 not found: ID does not exist"
Oct 31 19:43:56.169605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.169573    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"]
Oct 31 19:43:56.177018 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.176988    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-dg5wd"]
Oct 31 19:43:56.912493 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:43:56.911787    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" path="/var/lib/kubelet/pods/7315b844-4c65-4097-9656-ed99e582ce4c/volumes"
Oct 31 19:44:10.748282 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:10.748210    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" containerID="cri-o://fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960" gracePeriod=45
Oct 31 19:44:11.013388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.013324    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 19:44:11.100273 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.100236    5551 generic.go:356] "Generic (PLEG): container finished" podID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerID="fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960" exitCode=0
Oct 31 19:44:11.100425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.100283    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"
Oct 31 19:44:11.100425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.100290    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" event={"ID":"64915137-cd6c-43b7-b45b-c0b958e121a8","Type":"ContainerDied","Data":"fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960"}
Oct 31 19:44:11.100425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.100328    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" event={"ID":"64915137-cd6c-43b7-b45b-c0b958e121a8","Type":"ContainerDied","Data":"a51ee6741c0275d7c107df6ffece1b1ac7cbeb55766023697caaaf5527357bf6"}
Oct 31 19:44:11.100425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.100351    5551 scope.go:122] "RemoveContainer" containerID="fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960"
Oct 31 19:44:11.109995 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.109958    5551 scope.go:122] "RemoveContainer" containerID="fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960"
Oct 31 19:44:11.110244 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:44:11.110224    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960\": container with ID starting with fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960 not found: ID does not exist" containerID="fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960"
Oct 31 19:44:11.110285 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.110254    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960"} err="failed to get container status \"fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960\": rpc error: code = NotFound desc = could not find container \"fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960\": container with ID starting with fb22c601288d9c89e525ad0c173c3233924e285e4d8abc5ffdd85f6b24921960 not found: ID does not exist"
Oct 31 19:44:11.121444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.121415    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-model-cache\") pod \"64915137-cd6c-43b7-b45b-c0b958e121a8\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") "
Oct 31 19:44:11.121593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.121480    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-tmp-dir\") pod \"64915137-cd6c-43b7-b45b-c0b958e121a8\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") "
Oct 31 19:44:11.121593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.121518    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-dshm\") pod \"64915137-cd6c-43b7-b45b-c0b958e121a8\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") "
Oct 31 19:44:11.121593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.121569    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-home\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-home\") pod \"64915137-cd6c-43b7-b45b-c0b958e121a8\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") "
Oct 31 19:44:11.121769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.121640    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64915137-cd6c-43b7-b45b-c0b958e121a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/64915137-cd6c-43b7-b45b-c0b958e121a8-tls-certs\") pod \"64915137-cd6c-43b7-b45b-c0b958e121a8\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") "
Oct 31 19:44:11.121769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.121711    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-model-cache" pod "64915137-cd6c-43b7-b45b-c0b958e121a8" (UID: "64915137-cd6c-43b7-b45b-c0b958e121a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:11.121769 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.121756    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64915137-cd6c-43b7-b45b-c0b958e121a8-kube-api-access-gg9l8\" (UniqueName: \"kubernetes.io/projected/64915137-cd6c-43b7-b45b-c0b958e121a8-kube-api-access-gg9l8\") pod \"64915137-cd6c-43b7-b45b-c0b958e121a8\" (UID: \"64915137-cd6c-43b7-b45b-c0b958e121a8\") "
Oct 31 19:44:11.121924 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.121886    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-home" pod "64915137-cd6c-43b7-b45b-c0b958e121a8" (UID: "64915137-cd6c-43b7-b45b-c0b958e121a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:11.122112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.122085    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-tmp-dir" pod "64915137-cd6c-43b7-b45b-c0b958e121a8" (UID: "64915137-cd6c-43b7-b45b-c0b958e121a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:11.122231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.122209    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:11.122231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.122236    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:11.122231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.122251    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:11.123876 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.123851    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-dshm" pod "64915137-cd6c-43b7-b45b-c0b958e121a8" (UID: "64915137-cd6c-43b7-b45b-c0b958e121a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:11.124001 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.123983    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64915137-cd6c-43b7-b45b-c0b958e121a8-kube-api-access-gg9l8" pod "64915137-cd6c-43b7-b45b-c0b958e121a8" (UID: "64915137-cd6c-43b7-b45b-c0b958e121a8"). InnerVolumeSpecName "kube-api-access-gg9l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:11.124082 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.124067    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64915137-cd6c-43b7-b45b-c0b958e121a8-tls-certs" pod "64915137-cd6c-43b7-b45b-c0b958e121a8" (UID: "64915137-cd6c-43b7-b45b-c0b958e121a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:44:11.229406 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.224498    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg9l8\" (UniqueName: \"kubernetes.io/projected/64915137-cd6c-43b7-b45b-c0b958e121a8-kube-api-access-gg9l8\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:11.229406 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.224535    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64915137-cd6c-43b7-b45b-c0b958e121a8-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:11.229406 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.224563    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64915137-cd6c-43b7-b45b-c0b958e121a8-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:11.544117 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.544080    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"]
Oct 31 19:44:11.574352 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.574257    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l"]
Oct 31 19:44:11.990578 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:11.990529    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-88085e/distinct-w2-88085e-kserve-dbb778945-9rf8l" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:44:12.908894 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:12.908803    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" path="/var/lib/kubelet/pods/64915137-cd6c-43b7-b45b-c0b958e121a8/volumes"
Oct 31 19:44:16.604569 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:44:16.604529    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:44:25.499980 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.499942    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"]
Oct 31 19:44:25.500428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500387    5551 cpu_manager.go:414] "containerMap: removing container" podUID="abfb49ee-7672-4e09-be02-f0dc4aa237de" containerName="main"
Oct 31 19:44:25.500428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500399    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfb49ee-7672-4e09-be02-f0dc4aa237de" containerName="main"
Oct 31 19:44:25.500428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500421    5551 cpu_manager.go:414] "containerMap: removing container" podUID="88b55310-85c9-403e-98de-4be997a7aeaf" containerName="main"
Oct 31 19:44:25.500428 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500426    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b55310-85c9-403e-98de-4be997a7aeaf" containerName="main"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500433    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500438    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500443    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500448    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500474    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b1993cb5-c116-4c47-be36-3893da5088c9" containerName="maas-api"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500479    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1993cb5-c116-4c47-be36-3893da5088c9" containerName="maas-api"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500488    5551 cpu_manager.go:414] "containerMap: removing container" podUID="46532445-f525-4dfd-ac76-c0d2e7c26f2d" containerName="payload-pre-processing"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500493    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46532445-f525-4dfd-ac76-c0d2e7c26f2d" containerName="payload-pre-processing"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500501    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500506    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500517    5551 cpu_manager.go:414] "containerMap: removing container" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500522    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500530    5551 cpu_manager.go:414] "containerMap: removing container" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500534    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500542    5551 cpu_manager.go:414] "containerMap: removing container" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500547    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500555    5551 cpu_manager.go:414] "containerMap: removing container" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500561    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500663    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b869d2ac-8f69-4e85-8669-814ea8927d62" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500676    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfb49ee-7672-4e09-be02-f0dc4aa237de" containerName="main"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500688    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b55310-85c9-403e-98de-4be997a7aeaf" containerName="main"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500694    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500699    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7ec9b1-6584-4840-a1b7-8b4ae5f2a85a" containerName="istio-proxy"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500707    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1993cb5-c116-4c47-be36-3893da5088c9" containerName="maas-api"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500713    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="46532445-f525-4dfd-ac76-c0d2e7c26f2d" containerName="payload-pre-processing"
Oct 31 19:44:25.500706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500720    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy"
Oct 31 19:44:25.501668 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500726    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="64915137-cd6c-43b7-b45b-c0b958e121a8" containerName="main"
Oct 31 19:44:25.501668 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.500732    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e4dd78-4759-4bdb-9523-33ad3808763f" containerName="istio-proxy"
Oct 31 19:44:25.504205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.504188    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.556277 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.556256    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-88085e-openshift-default-dockercfg-96c8d\""
Oct 31 19:44:25.589914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.589883    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3c415-01d2-4164-874c-7962a1f11af3-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.590020 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.589935    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.590064 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.590048    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.590107 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.590076    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.590143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.590107    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbl28\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-kube-api-access-nbl28\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.590204 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.590190    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.590241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.590211    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.590241 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.590230    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.590364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.590350    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3c415-01d2-4164-874c-7962a1f11af3-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.608228 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.608207    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"]
Oct 31 19:44:25.691667 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691638    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.691834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691672    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.691834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691703    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbl28\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-kube-api-access-nbl28\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.691834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691742    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.691834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691765    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.691834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691795    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.692120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691879    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3c415-01d2-4164-874c-7962a1f11af3-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.692120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691936    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3c415-01d2-4164-874c-7962a1f11af3-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.692120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.691998    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.692120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.692069    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.692341 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.692195    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.692341 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.692279    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.692654 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.692356    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.692724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.692660    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3c415-01d2-4164-874c-7962a1f11af3-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.694190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.694166    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.694388 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.694372    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3c415-01d2-4164-874c-7962a1f11af3-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.754355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.751291    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.754355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.751557    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbl28\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-kube-api-access-nbl28\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:25.817056 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:25.817022    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:26.249860 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:26.249833    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"]
Oct 31 19:44:26.251353 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:44:26.251326    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d3c415_01d2_4164_874c_7962a1f11af3.slice/crio-820f94a622f02ffbcd263fba4488edf3961c774567f5ec23e01631359a663d72 WatchSource:0}: Error finding container 820f94a622f02ffbcd263fba4488edf3961c774567f5ec23e01631359a663d72: Status 404 returned error can't find the container with id 820f94a622f02ffbcd263fba4488edf3961c774567f5ec23e01631359a663d72
Oct 31 19:44:26.253616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:26.253584    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:44:26.253678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:26.253652    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:44:26.253723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:26.253680    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:44:27.201008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:27.200974    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" event={"ID":"48d3c415-01d2-4164-874c-7962a1f11af3","Type":"ContainerStarted","Data":"8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f"}
Oct 31 19:44:27.201448 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:27.201023    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" event={"ID":"48d3c415-01d2-4164-874c-7962a1f11af3","Type":"ContainerStarted","Data":"820f94a622f02ffbcd263fba4488edf3961c774567f5ec23e01631359a663d72"}
Oct 31 19:44:27.393492 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:27.393418    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" podStartSLOduration=2.393402003 podStartE2EDuration="2.393402003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:27.391442876 +0000 UTC m=+4970.622681167" watchObservedRunningTime="2026-10-31 19:44:27.393402003 +0000 UTC m=+4970.624640271"
Oct 31 19:44:27.818101 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:27.818065    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:27.819574 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:27.819545    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:44:27.819722 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:27.819604    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 19:44:28.818219 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:28.818174    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:44:28.818760 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:28.818238    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 19:44:29.817442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:29.817389    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:44:29.817670 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:29.817530    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 19:44:30.818051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:30.818016    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:44:30.818502 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:30.818070    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 19:44:31.817545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:31.817501    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:44:31.817719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:31.817563    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 19:44:32.818182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:32.818146    5551 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:44:32.818644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:32.818202    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 19:44:34.772839 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:34.772803    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:34.773834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:34.773806    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:34.774724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:34.774702    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:37.045821 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:37.045792    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"]
Oct 31 19:44:37.095305 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:37.095272    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe7ed9e-4js9x"]
Oct 31 19:44:37.705583 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:37.705541    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"]
Oct 31 19:44:37.705785 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:37.705761    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n" podUID="4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" containerName="maas-api" containerID="cri-o://bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb" gracePeriod=30
Oct 31 19:44:37.976378 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:37.976341    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 19:44:38.070314 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.070275    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-metrics-certs\") pod \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") "
Oct 31 19:44:38.070837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.070397    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-kube-api-access-8zl94\" (UniqueName: \"kubernetes.io/projected/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-kube-api-access-8zl94\") pod \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") "
Oct 31 19:44:38.070837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.070443    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls\") pod \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") "
Oct 31 19:44:38.070837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.070615    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-openshift-service-ca\") pod \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\" (UID: \"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d\") "
Oct 31 19:44:38.071122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.071088    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-openshift-service-ca" pod "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" (UID: "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:38.072910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.072880    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls" pod "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" (UID: "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:44:38.073158 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.073138    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-metrics-certs" pod "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" (UID: "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:44:38.073229 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.073186    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-kube-api-access-8zl94" pod "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" (UID: "4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d"). InnerVolumeSpecName "kube-api-access-8zl94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:38.172351 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.172320    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:38.172351 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.172350    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zl94\" (UniqueName: \"kubernetes.io/projected/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-kube-api-access-8zl94\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:38.172594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.172360    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:38.172594 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.172371    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:38.274870 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.274789    5551 generic.go:356] "Generic (PLEG): container finished" podID="4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" containerID="bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb" exitCode=0
Oct 31 19:44:38.274870 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.274828    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n" event={"ID":"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d","Type":"ContainerDied","Data":"bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb"}
Oct 31 19:44:38.274870 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.274838    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"
Oct 31 19:44:38.274870 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.274868    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n" event={"ID":"4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d","Type":"ContainerDied","Data":"6f0fd24b296e858cce2dfd1cded27997b282d25cbfe15e495a7a3dd3de2564d1"}
Oct 31 19:44:38.275179 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.274891    5551 scope.go:122] "RemoveContainer" containerID="bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb"
Oct 31 19:44:38.286476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.286445    5551 scope.go:122] "RemoveContainer" containerID="bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb"
Oct 31 19:44:38.286803 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:44:38.286783    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb\": container with ID starting with bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb not found: ID does not exist" containerID="bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb"
Oct 31 19:44:38.286874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.286812    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb"} err="failed to get container status \"bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb\": rpc error: code = NotFound desc = could not find container \"bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb\": container with ID starting with bd29a0b118d24511ba6869c20d449d04168763f54e830af898b0cc1c8ef255cb not found: ID does not exist"
Oct 31 19:44:38.438994 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.438962    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"]
Oct 31 19:44:38.441104 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.441079    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-88085e-9fb45555b-dww8n"]
Oct 31 19:44:38.909773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.909739    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" path="/var/lib/kubelet/pods/4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d/volumes"
Oct 31 19:44:38.910299 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:38.910278    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f757acab-a38f-4d08-a965-bf5e26f4ab6b" path="/var/lib/kubelet/pods/f757acab-a38f-4d08-a965-bf5e26f4ab6b/volumes"
Oct 31 19:44:46.608061 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:44:46.608018    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:44:50.716261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:50.716226    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"]
Oct 31 19:44:50.716821 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:50.716537    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy" containerID="cri-o://8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f" gracePeriod=30
Oct 31 19:44:55.962274 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:55.962245    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:56.083332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083264    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-certs\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083332 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083311    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-istio-token\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-istio-token\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083355    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d3c415-01d2-4164-874c-7962a1f11af3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3c415-01d2-4164-874c-7962a1f11af3-istiod-ca-cert\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083392    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-envoy\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083438    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-socket\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083526    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-credential-socket\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083563    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48d3c415-01d2-4164-874c-7962a1f11af3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3c415-01d2-4164-874c-7962a1f11af3-istio-podinfo\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083572    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-certs" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.083912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083647    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-kube-api-access-nbl28\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-kube-api-access-nbl28\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083705    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-data\") pod \"48d3c415-01d2-4164-874c-7962a1f11af3\" (UID: \"48d3c415-01d2-4164-874c-7962a1f11af3\") "
Oct 31 19:44:56.083912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083790    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-socket" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.083912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.083803    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d3c415-01d2-4164-874c-7962a1f11af3-istiod-ca-cert" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:56.084134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.084012    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-credential-socket" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.084134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.084030    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-data" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.084666 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.084216    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.084666 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.084319    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3c415-01d2-4164-874c-7962a1f11af3-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.084666 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.084333    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.084666 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.084347    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.084666 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.084360    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.085748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.085726    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-envoy" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.086002 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.085978    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-kube-api-access-nbl28" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "kube-api-access-nbl28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:56.086294 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.086273    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48d3c415-01d2-4164-874c-7962a1f11af3-istio-podinfo" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:44:56.086357 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.086297    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-istio-token" pod "48d3c415-01d2-4164-874c-7962a1f11af3" (UID: "48d3c415-01d2-4164-874c-7962a1f11af3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:56.185035 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.185000    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbl28\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-kube-api-access-nbl28\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.185035 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.185033    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3c415-01d2-4164-874c-7962a1f11af3-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.185035 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.185045    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3c415-01d2-4164-874c-7962a1f11af3-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.185279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.185054    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3c415-01d2-4164-874c-7962a1f11af3-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.385001 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.384971    5551 generic.go:356] "Generic (PLEG): container finished" podID="48d3c415-01d2-4164-874c-7962a1f11af3" containerID="8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f" exitCode=0
Oct 31 19:44:56.385155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.385020    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"
Oct 31 19:44:56.385155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.385065    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" event={"ID":"48d3c415-01d2-4164-874c-7962a1f11af3","Type":"ContainerDied","Data":"8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f"}
Oct 31 19:44:56.385155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.385124    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr" event={"ID":"48d3c415-01d2-4164-874c-7962a1f11af3","Type":"ContainerDied","Data":"820f94a622f02ffbcd263fba4488edf3961c774567f5ec23e01631359a663d72"}
Oct 31 19:44:56.385155 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.385145    5551 scope.go:122] "RemoveContainer" containerID="8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f"
Oct 31 19:44:56.395793 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.395775    5551 scope.go:122] "RemoveContainer" containerID="8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f"
Oct 31 19:44:56.396050 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:44:56.396034    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f\": container with ID starting with 8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f not found: ID does not exist" containerID="8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f"
Oct 31 19:44:56.396091 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.396058    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f"} err="failed to get container status \"8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f\": rpc error: code = NotFound desc = could not find container \"8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f\": container with ID starting with 8e0fb2b5c610b57af1406262fa643af95f741325573e4180f1054227077d6c0f not found: ID does not exist"
Oct 31 19:44:56.548281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.548248    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"]
Oct 31 19:44:56.597627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.597592    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-2kvnr"]
Oct 31 19:44:56.909688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:44:56.909653    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" path="/var/lib/kubelet/pods/48d3c415-01d2-4164-874c-7962a1f11af3/volumes"
Oct 31 19:45:00.162177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.162144    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"]
Oct 31 19:45:00.162726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.162690    5551 cpu_manager.go:414] "containerMap: removing container" podUID="4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" containerName="maas-api"
Oct 31 19:45:00.162726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.162706    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" containerName="maas-api"
Oct 31 19:45:00.162726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.162720    5551 cpu_manager.go:414] "containerMap: removing container" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy"
Oct 31 19:45:00.162726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.162725    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy"
Oct 31 19:45:00.162896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.162801    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bee3783-8dcd-42c6-86aa-abdd7ff0ae8d" containerName="maas-api"
Oct 31 19:45:00.162896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.162810    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d3c415-01d2-4164-874c-7962a1f11af3" containerName="istio-proxy"
Oct 31 19:45:00.162896 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.162822    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="7315b844-4c65-4097-9656-ed99e582ce4c" containerName="istio-proxy"
Oct 31 19:45:00.167286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.167268    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:00.218906 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.218879    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:45:00.228342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.228313    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe4c64f-6891-4be8-9021-33a63ef9d722-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-llpkl\" (UID: \"5fe4c64f-6891-4be8-9021-33a63ef9d722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:00.228499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.228371    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvd7m\" (UniqueName: \"kubernetes.io/projected/5fe4c64f-6891-4be8-9021-33a63ef9d722-kube-api-access-kvd7m\") pod \"maas-api-key-cleanup-29891265-llpkl\" (UID: \"5fe4c64f-6891-4be8-9021-33a63ef9d722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:00.270849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.270822    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"]
Oct 31 19:45:00.329770 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.329736    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe4c64f-6891-4be8-9021-33a63ef9d722-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-llpkl\" (UID: \"5fe4c64f-6891-4be8-9021-33a63ef9d722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:00.329928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.329819    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvd7m\" (UniqueName: \"kubernetes.io/projected/5fe4c64f-6891-4be8-9021-33a63ef9d722-kube-api-access-kvd7m\") pod \"maas-api-key-cleanup-29891265-llpkl\" (UID: \"5fe4c64f-6891-4be8-9021-33a63ef9d722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:00.330408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.330390    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe4c64f-6891-4be8-9021-33a63ef9d722-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-llpkl\" (UID: \"5fe4c64f-6891-4be8-9021-33a63ef9d722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:00.389597 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.389572    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvd7m\" (UniqueName: \"kubernetes.io/projected/5fe4c64f-6891-4be8-9021-33a63ef9d722-kube-api-access-kvd7m\") pod \"maas-api-key-cleanup-29891265-llpkl\" (UID: \"5fe4c64f-6891-4be8-9021-33a63ef9d722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:00.480805 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.480729    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:00.859401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:00.859376    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"]
Oct 31 19:45:00.861139 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:45:00.861109    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe4c64f_6891_4be8_9021_33a63ef9d722.slice/crio-9a5bd682495b5ce3585ac1a5e75750ae4d41309d2b0af1e2b605964a252e72f6 WatchSource:0}: Error finding container 9a5bd682495b5ce3585ac1a5e75750ae4d41309d2b0af1e2b605964a252e72f6: Status 404 returned error can't find the container with id 9a5bd682495b5ce3585ac1a5e75750ae4d41309d2b0af1e2b605964a252e72f6
Oct 31 19:45:01.426596 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:01.426484    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" event={"ID":"5fe4c64f-6891-4be8-9021-33a63ef9d722","Type":"ContainerStarted","Data":"3c0e0e72f10e26780052ef5d67d7b4ea756d6cfeca3482cfe105fae4e92e4f7d"}
Oct 31 19:45:01.426596 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:01.426574    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" event={"ID":"5fe4c64f-6891-4be8-9021-33a63ef9d722","Type":"ContainerStarted","Data":"9a5bd682495b5ce3585ac1a5e75750ae4d41309d2b0af1e2b605964a252e72f6"}
Oct 31 19:45:01.548796 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:01.548741    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" podStartSLOduration=1.548725602 podStartE2EDuration="1.548725602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:01.547431016 +0000 UTC m=+5004.778669307" watchObservedRunningTime="2026-10-31 19:45:01.548725602 +0000 UTC m=+5004.779963894"
Oct 31 19:45:02.433845 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:02.433807    5551 generic.go:356] "Generic (PLEG): container finished" podID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerID="3c0e0e72f10e26780052ef5d67d7b4ea756d6cfeca3482cfe105fae4e92e4f7d" exitCode=60
Oct 31 19:45:02.434262 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:02.433881    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" event={"ID":"5fe4c64f-6891-4be8-9021-33a63ef9d722","Type":"ContainerDied","Data":"3c0e0e72f10e26780052ef5d67d7b4ea756d6cfeca3482cfe105fae4e92e4f7d"}
Oct 31 19:45:02.434262 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:02.434109    5551 scope.go:122] "RemoveContainer" containerID="3c0e0e72f10e26780052ef5d67d7b4ea756d6cfeca3482cfe105fae4e92e4f7d"
Oct 31 19:45:03.441353 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:03.441320    5551 generic.go:356] "Generic (PLEG): container finished" podID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerID="3c1c9178a926997b6adab004dfba30e518dcbb30f8ce849d41c6a63686ad642e" exitCode=60
Oct 31 19:45:03.441838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:03.441359    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" event={"ID":"5fe4c64f-6891-4be8-9021-33a63ef9d722","Type":"ContainerDied","Data":"3c1c9178a926997b6adab004dfba30e518dcbb30f8ce849d41c6a63686ad642e"}
Oct 31 19:45:03.441838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:03.441411    5551 scope.go:122] "RemoveContainer" containerID="3c0e0e72f10e26780052ef5d67d7b4ea756d6cfeca3482cfe105fae4e92e4f7d"
Oct 31 19:45:03.441838 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:03.441665    5551 scope.go:122] "RemoveContainer" containerID="3c1c9178a926997b6adab004dfba30e518dcbb30f8ce849d41c6a63686ad642e"
Oct 31 19:45:03.441983 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:45:03.441938    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-llpkl_odh-ai-gateway-infra(5fe4c64f-6891-4be8-9021-33a63ef9d722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722"
Oct 31 19:45:04.448892 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:04.448864    5551 scope.go:122] "RemoveContainer" containerID="3c1c9178a926997b6adab004dfba30e518dcbb30f8ce849d41c6a63686ad642e"
Oct 31 19:45:04.449293 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:45:04.449108    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-llpkl_odh-ai-gateway-infra(5fe4c64f-6891-4be8-9021-33a63ef9d722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722"
Oct 31 19:45:16.604700 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:45:16.604657    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:45:16.905443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:16.905412    5551 scope.go:122] "RemoveContainer" containerID="3c1c9178a926997b6adab004dfba30e518dcbb30f8ce849d41c6a63686ad642e"
Oct 31 19:45:17.532758 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:17.532675    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" event={"ID":"5fe4c64f-6891-4be8-9021-33a63ef9d722","Type":"ContainerStarted","Data":"562dd9dec478777f4283ca753cb503b6fd9fef4c72f8b0bfacb13d856cc882cc"}
Oct 31 19:45:18.542063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:18.542027    5551 generic.go:356] "Generic (PLEG): container finished" podID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerID="562dd9dec478777f4283ca753cb503b6fd9fef4c72f8b0bfacb13d856cc882cc" exitCode=60
Oct 31 19:45:18.542600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:18.542092    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" event={"ID":"5fe4c64f-6891-4be8-9021-33a63ef9d722","Type":"ContainerDied","Data":"562dd9dec478777f4283ca753cb503b6fd9fef4c72f8b0bfacb13d856cc882cc"}
Oct 31 19:45:18.542600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:18.542151    5551 scope.go:122] "RemoveContainer" containerID="3c1c9178a926997b6adab004dfba30e518dcbb30f8ce849d41c6a63686ad642e"
Oct 31 19:45:18.542600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:18.542536    5551 scope.go:122] "RemoveContainer" containerID="562dd9dec478777f4283ca753cb503b6fd9fef4c72f8b0bfacb13d856cc882cc"
Oct 31 19:45:18.542822 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:45:18.542798    5551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-llpkl_odh-ai-gateway-infra(5fe4c64f-6891-4be8-9021-33a63ef9d722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722"
Oct 31 19:45:18.686343 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:18.686304    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"]
Oct 31 19:45:19.690137 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:19.690112    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:19.741009 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:19.740983    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe4c64f-6891-4be8-9021-33a63ef9d722-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe4c64f-6891-4be8-9021-33a63ef9d722-maas-api-service-ca\") pod \"5fe4c64f-6891-4be8-9021-33a63ef9d722\" (UID: \"5fe4c64f-6891-4be8-9021-33a63ef9d722\") "
Oct 31 19:45:19.741132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:19.741104    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe4c64f-6891-4be8-9021-33a63ef9d722-kube-api-access-kvd7m\" (UniqueName: \"kubernetes.io/projected/5fe4c64f-6891-4be8-9021-33a63ef9d722-kube-api-access-kvd7m\") pod \"5fe4c64f-6891-4be8-9021-33a63ef9d722\" (UID: \"5fe4c64f-6891-4be8-9021-33a63ef9d722\") "
Oct 31 19:45:19.741382 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:19.741360    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe4c64f-6891-4be8-9021-33a63ef9d722-maas-api-service-ca" pod "5fe4c64f-6891-4be8-9021-33a63ef9d722" (UID: "5fe4c64f-6891-4be8-9021-33a63ef9d722"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:19.741562 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:19.741548    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe4c64f-6891-4be8-9021-33a63ef9d722-maas-api-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:19.743227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:19.743202    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe4c64f-6891-4be8-9021-33a63ef9d722-kube-api-access-kvd7m" pod "5fe4c64f-6891-4be8-9021-33a63ef9d722" (UID: "5fe4c64f-6891-4be8-9021-33a63ef9d722"). InnerVolumeSpecName "kube-api-access-kvd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:19.842292 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:19.842218    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvd7m\" (UniqueName: \"kubernetes.io/projected/5fe4c64f-6891-4be8-9021-33a63ef9d722-kube-api-access-kvd7m\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:20.558175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:20.558139    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"
Oct 31 19:45:20.558175 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:20.558163    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl" event={"ID":"5fe4c64f-6891-4be8-9021-33a63ef9d722","Type":"ContainerDied","Data":"9a5bd682495b5ce3585ac1a5e75750ae4d41309d2b0af1e2b605964a252e72f6"}
Oct 31 19:45:20.558424 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:20.558232    5551 scope.go:122] "RemoveContainer" containerID="562dd9dec478777f4283ca753cb503b6fd9fef4c72f8b0bfacb13d856cc882cc"
Oct 31 19:45:20.726197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:20.726153    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"]
Oct 31 19:45:20.732930 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:20.732896    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-llpkl"]
Oct 31 19:45:20.908044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:20.908014    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" path="/var/lib/kubelet/pods/5fe4c64f-6891-4be8-9021-33a63ef9d722/volumes"
Oct 31 19:45:40.814974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:40.814933    5551 scope.go:122] "RemoveContainer" containerID="66eff50b86bab034b39d95adcb98f62170d60408959b0ae0f8079ad074918145"
Oct 31 19:45:46.604072 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:45:46.603986    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:45:52.566639 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.566600    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"]
Oct 31 19:45:52.567286 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.566861    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx" podUID="ee395a46-1e79-45e8-948f-b714877ff218" containerName="maas-api" containerID="cri-o://5192d7f1694b548cb52631eb5da41fcf558760e1169a4c4ccd61fb6dcbdece14" gracePeriod=30
Oct 31 19:45:52.759053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.758866    5551 generic.go:356] "Generic (PLEG): container finished" podID="ee395a46-1e79-45e8-948f-b714877ff218" containerID="5192d7f1694b548cb52631eb5da41fcf558760e1169a4c4ccd61fb6dcbdece14" exitCode=0
Oct 31 19:45:52.759053 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.758974    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx" event={"ID":"ee395a46-1e79-45e8-948f-b714877ff218","Type":"ContainerDied","Data":"5192d7f1694b548cb52631eb5da41fcf558760e1169a4c4ccd61fb6dcbdece14"}
Oct 31 19:45:52.819434 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.819370    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:45:52.895960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.895934    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee395a46-1e79-45e8-948f-b714877ff218-kube-api-access-cqp7c\" (UniqueName: \"kubernetes.io/projected/ee395a46-1e79-45e8-948f-b714877ff218-kube-api-access-cqp7c\") pod \"ee395a46-1e79-45e8-948f-b714877ff218\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") "
Oct 31 19:45:52.896103 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.896007    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls\") pod \"ee395a46-1e79-45e8-948f-b714877ff218\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") "
Oct 31 19:45:52.896147 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.896130    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-metrics-certs\") pod \"ee395a46-1e79-45e8-948f-b714877ff218\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") "
Oct 31 19:45:52.896218 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.896201    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee395a46-1e79-45e8-948f-b714877ff218-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee395a46-1e79-45e8-948f-b714877ff218-openshift-service-ca\") pod \"ee395a46-1e79-45e8-948f-b714877ff218\" (UID: \"ee395a46-1e79-45e8-948f-b714877ff218\") "
Oct 31 19:45:52.896585 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.896560    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee395a46-1e79-45e8-948f-b714877ff218-openshift-service-ca" pod "ee395a46-1e79-45e8-948f-b714877ff218" (UID: "ee395a46-1e79-45e8-948f-b714877ff218"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:52.898138 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.898119    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls" pod "ee395a46-1e79-45e8-948f-b714877ff218" (UID: "ee395a46-1e79-45e8-948f-b714877ff218"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:45:52.898206 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.898157    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee395a46-1e79-45e8-948f-b714877ff218-kube-api-access-cqp7c" pod "ee395a46-1e79-45e8-948f-b714877ff218" (UID: "ee395a46-1e79-45e8-948f-b714877ff218"). InnerVolumeSpecName "kube-api-access-cqp7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:52.898361 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.898347    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-metrics-certs" pod "ee395a46-1e79-45e8-948f-b714877ff218" (UID: "ee395a46-1e79-45e8-948f-b714877ff218"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:45:52.998023 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.997969    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:52.998023 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.998015    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee395a46-1e79-45e8-948f-b714877ff218-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:52.998023 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.998031    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqp7c\" (UniqueName: \"kubernetes.io/projected/ee395a46-1e79-45e8-948f-b714877ff218-kube-api-access-cqp7c\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:52.998285 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:52.998045    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee395a46-1e79-45e8-948f-b714877ff218-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:53.766392 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:53.766356    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"
Oct 31 19:45:53.766873 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:53.766389    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx" event={"ID":"ee395a46-1e79-45e8-948f-b714877ff218","Type":"ContainerDied","Data":"75b8514bffd4adb90e4429ea384c8c8ce93405d82fcbeee71f4d85d8589b9a99"}
Oct 31 19:45:53.766873 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:53.766488    5551 scope.go:122] "RemoveContainer" containerID="5192d7f1694b548cb52631eb5da41fcf558760e1169a4c4ccd61fb6dcbdece14"
Oct 31 19:45:53.932031 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:53.931994    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"]
Oct 31 19:45:53.981196 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:53.981171    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0f794f-79cbd6d7c6-6h4fx"]
Oct 31 19:45:54.910237 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:45:54.910204    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee395a46-1e79-45e8-948f-b714877ff218" path="/var/lib/kubelet/pods/ee395a46-1e79-45e8-948f-b714877ff218/volumes"
Oct 31 19:46:09.114651 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.114610    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"]
Oct 31 19:46:09.115475 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.115385    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx" podUID="555fbf73-7e06-47c8-86ac-4e2236813469" containerName="maas-api" containerID="cri-o://e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977" gracePeriod=30
Oct 31 19:46:09.381443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.381417    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:46:09.473063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.465923    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls\") pod \"555fbf73-7e06-47c8-86ac-4e2236813469\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") "
Oct 31 19:46:09.473063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.466046    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555fbf73-7e06-47c8-86ac-4e2236813469-kube-api-access-mjrzd\" (UniqueName: \"kubernetes.io/projected/555fbf73-7e06-47c8-86ac-4e2236813469-kube-api-access-mjrzd\") pod \"555fbf73-7e06-47c8-86ac-4e2236813469\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") "
Oct 31 19:46:09.473063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.466099    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555fbf73-7e06-47c8-86ac-4e2236813469-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555fbf73-7e06-47c8-86ac-4e2236813469-openshift-service-ca\") pod \"555fbf73-7e06-47c8-86ac-4e2236813469\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") "
Oct 31 19:46:09.473063 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.466126    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-metrics-certs\") pod \"555fbf73-7e06-47c8-86ac-4e2236813469\" (UID: \"555fbf73-7e06-47c8-86ac-4e2236813469\") "
Oct 31 19:46:09.473448 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.473079    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-metrics-certs" pod "555fbf73-7e06-47c8-86ac-4e2236813469" (UID: "555fbf73-7e06-47c8-86ac-4e2236813469"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:09.473448 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.473437    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555fbf73-7e06-47c8-86ac-4e2236813469-openshift-service-ca" pod "555fbf73-7e06-47c8-86ac-4e2236813469" (UID: "555fbf73-7e06-47c8-86ac-4e2236813469"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:09.475631 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.475095    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls" pod "555fbf73-7e06-47c8-86ac-4e2236813469" (UID: "555fbf73-7e06-47c8-86ac-4e2236813469"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:09.477416 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.477380    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555fbf73-7e06-47c8-86ac-4e2236813469-kube-api-access-mjrzd" pod "555fbf73-7e06-47c8-86ac-4e2236813469" (UID: "555fbf73-7e06-47c8-86ac-4e2236813469"). InnerVolumeSpecName "kube-api-access-mjrzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:09.567724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.567692    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:09.567724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.567722    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrzd\" (UniqueName: \"kubernetes.io/projected/555fbf73-7e06-47c8-86ac-4e2236813469-kube-api-access-mjrzd\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:09.567919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.567732    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555fbf73-7e06-47c8-86ac-4e2236813469-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:09.567919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.567741    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fbf73-7e06-47c8-86ac-4e2236813469-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:09.861984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.861903    5551 generic.go:356] "Generic (PLEG): container finished" podID="555fbf73-7e06-47c8-86ac-4e2236813469" containerID="e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977" exitCode=0
Oct 31 19:46:09.861984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.861950    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"
Oct 31 19:46:09.861984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.861959    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx" event={"ID":"555fbf73-7e06-47c8-86ac-4e2236813469","Type":"ContainerDied","Data":"e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977"}
Oct 31 19:46:09.862290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.862000    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx" event={"ID":"555fbf73-7e06-47c8-86ac-4e2236813469","Type":"ContainerDied","Data":"793745d6cd19bd011c9a56c6a21277f1ec4a5789335862e21668e5c73234b006"}
Oct 31 19:46:09.862290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.862022    5551 scope.go:122] "RemoveContainer" containerID="e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977"
Oct 31 19:46:09.871718 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.871700    5551 scope.go:122] "RemoveContainer" containerID="e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977"
Oct 31 19:46:09.871966 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:46:09.871949    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977\": container with ID starting with e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977 not found: ID does not exist" containerID="e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977"
Oct 31 19:46:09.872016 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:09.871974    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977"} err="failed to get container status \"e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977\": rpc error: code = NotFound desc = could not find container \"e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977\": container with ID starting with e506484bac8e3f29939be231acecaabc0ad923dd0ccf469bf4b80a26bb213977 not found: ID does not exist"
Oct 31 19:46:10.050480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:10.046556    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"]
Oct 31 19:46:10.050480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:10.049312    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0f0e4e-c7b7b6667-pfcnx"]
Oct 31 19:46:10.907884 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:10.907855    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555fbf73-7e06-47c8-86ac-4e2236813469" path="/var/lib/kubelet/pods/555fbf73-7e06-47c8-86ac-4e2236813469/volumes"
Oct 31 19:46:13.494802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:13.494765    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"]
Oct 31 19:46:13.495219 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:13.494999    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy" containerID="cri-o://c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3" gracePeriod=30
Oct 31 19:46:15.064849 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.064817    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"]
Oct 31 19:46:15.065375 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065357    5551 cpu_manager.go:414] "containerMap: removing container" podUID="ee395a46-1e79-45e8-948f-b714877ff218" containerName="maas-api"
Oct 31 19:46:15.065442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065377    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee395a46-1e79-45e8-948f-b714877ff218" containerName="maas-api"
Oct 31 19:46:15.065442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065386    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:15.065442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065392    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:15.065442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065425    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:15.065442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065430    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:15.065442 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065442    5551 cpu_manager.go:414] "containerMap: removing container" podUID="555fbf73-7e06-47c8-86ac-4e2236813469" containerName="maas-api"
Oct 31 19:46:15.065662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065447    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555fbf73-7e06-47c8-86ac-4e2236813469" containerName="maas-api"
Oct 31 19:46:15.065662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065475    5551 cpu_manager.go:414] "containerMap: removing container" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:15.065662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065480    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:15.065662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065558    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:15.065662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065566    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="555fbf73-7e06-47c8-86ac-4e2236813469" containerName="maas-api"
Oct 31 19:46:15.065662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065574    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:15.065662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.065584    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee395a46-1e79-45e8-948f-b714877ff218" containerName="maas-api"
Oct 31 19:46:15.070020 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.070003    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.125845 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.125811    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"]
Oct 31 19:46:15.172471 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.172428    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:46:15.172661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.172509    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:46:15.221863 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.221832    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:46:15.221863 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.221832    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:46:15.232476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.232430    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.232644 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.232566    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.232731 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.232653    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xgt\" (UniqueName: \"kubernetes.io/projected/781a1a0e-264a-49c6-abcb-4255d46a8e5b-kube-api-access-r9xgt\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.333436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.333340    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xgt\" (UniqueName: \"kubernetes.io/projected/781a1a0e-264a-49c6-abcb-4255d46a8e5b-kube-api-access-r9xgt\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.333436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.333413    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.333680 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.333552    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.334223 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.334190    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.336115 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.336090    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.390405 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.390364    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xgt\" (UniqueName: \"kubernetes.io/projected/781a1a0e-264a-49c6-abcb-4255d46a8e5b-kube-api-access-r9xgt\") pod \"maas-api-revoke-sub-32b3a852-w88mr\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:15.683254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:15.683208    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:16.062437 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:16.062414    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"]
Oct 31 19:46:16.065087 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:46:16.065055    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781a1a0e_264a_49c6_abcb_4255d46a8e5b.slice/crio-67dded2e69a1a17bc014eb456edb821dfcf8b79b4c92bb44511d090d18e4ab78 WatchSource:0}: Error finding container 67dded2e69a1a17bc014eb456edb821dfcf8b79b4c92bb44511d090d18e4ab78: Status 404 returned error can't find the container with id 67dded2e69a1a17bc014eb456edb821dfcf8b79b4c92bb44511d090d18e4ab78
Oct 31 19:46:16.606294 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:46:16.604834    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:46:16.911205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:16.911166    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr" event={"ID":"781a1a0e-264a-49c6-abcb-4255d46a8e5b","Type":"ContainerStarted","Data":"98fa017c7a6f04b350381aee4946c0e3481a2e1347589c7d46d32e3325c2beef"}
Oct 31 19:46:16.911381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:16.911217    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr" event={"ID":"781a1a0e-264a-49c6-abcb-4255d46a8e5b","Type":"ContainerStarted","Data":"67dded2e69a1a17bc014eb456edb821dfcf8b79b4c92bb44511d090d18e4ab78"}
Oct 31 19:46:17.042084 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:17.042021    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr" podStartSLOduration=2.042006402 podStartE2EDuration="2.042006402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:17.039057431 +0000 UTC m=+5080.270295722" watchObservedRunningTime="2026-10-31 19:46:17.042006402 +0000 UTC m=+5080.273244691"
Oct 31 19:46:17.917617 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:17.917587    5551 generic.go:356] "Generic (PLEG): container finished" podID="781a1a0e-264a-49c6-abcb-4255d46a8e5b" containerID="98fa017c7a6f04b350381aee4946c0e3481a2e1347589c7d46d32e3325c2beef" exitCode=0
Oct 31 19:46:17.918027 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:17.917639    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr" event={"ID":"781a1a0e-264a-49c6-abcb-4255d46a8e5b","Type":"ContainerDied","Data":"98fa017c7a6f04b350381aee4946c0e3481a2e1347589c7d46d32e3325c2beef"}
Oct 31 19:46:18.746648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.746620    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:46:18.875414 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875383    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-istio-token\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-istio-token\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.875599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875446    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611474c7-427f-438e-83ed-257b09db4ff1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611474c7-427f-438e-83ed-257b09db4ff1-istiod-ca-cert\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.875599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875565    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-envoy\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.875730 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875597    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-kube-api-access-mmz7c\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-kube-api-access-mmz7c\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.875730 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875640    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-data\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.875730 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875669    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/611474c7-427f-438e-83ed-257b09db4ff1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611474c7-427f-438e-83ed-257b09db4ff1-istio-podinfo\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.875888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875752    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-certs\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.875888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875824    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-credential-socket\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.875888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875856    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-socket\") pod \"611474c7-427f-438e-83ed-257b09db4ff1\" (UID: \"611474c7-427f-438e-83ed-257b09db4ff1\") "
Oct 31 19:46:18.876044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875900    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-data" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:18.876044 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.875915    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611474c7-427f-438e-83ed-257b09db4ff1-istiod-ca-cert" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:18.876151 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.876101    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-credential-socket" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:18.876151 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.876112    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-certs" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:18.876292 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.876261    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/611474c7-427f-438e-83ed-257b09db4ff1-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:18.876360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.876298    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:18.876360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.876312    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:18.876360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.876328    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:18.876544 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.876516    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-socket" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:18.878238 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.878208    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-istio-token" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:18.878362 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.878261    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/611474c7-427f-438e-83ed-257b09db4ff1-istio-podinfo" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:46:18.878412 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.878376    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-kube-api-access-mmz7c" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "kube-api-access-mmz7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:18.878453 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.878439    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-envoy" pod "611474c7-427f-438e-83ed-257b09db4ff1" (UID: "611474c7-427f-438e-83ed-257b09db4ff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:18.926142 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.925441    5551 generic.go:356] "Generic (PLEG): container finished" podID="611474c7-427f-438e-83ed-257b09db4ff1" containerID="c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3" exitCode=0
Oct 31 19:46:18.926581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.925512    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" event={"ID":"611474c7-427f-438e-83ed-257b09db4ff1","Type":"ContainerDied","Data":"c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3"}
Oct 31 19:46:18.926581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.926249    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7" event={"ID":"611474c7-427f-438e-83ed-257b09db4ff1","Type":"ContainerDied","Data":"ab68f911df1df9469fea57ece9d41dd36e2262a21e2ad8884e8c6e248b90a314"}
Oct 31 19:46:18.926581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.926276    5551 scope.go:122] "RemoveContainer" containerID="c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3"
Oct 31 19:46:18.926581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.925546    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"
Oct 31 19:46:18.936233 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.936209    5551 scope.go:122] "RemoveContainer" containerID="1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97"
Oct 31 19:46:18.948042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.948015    5551 scope.go:122] "RemoveContainer" containerID="c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3"
Oct 31 19:46:18.948421 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:46:18.948396    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3\": container with ID starting with c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3 not found: ID does not exist" containerID="c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3"
Oct 31 19:46:18.948625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.948435    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3"} err="failed to get container status \"c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3\": rpc error: code = NotFound desc = could not find container \"c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3\": container with ID starting with c9a4aab890ced2454f5d8dbda414b438435e0fce9ca40398719076f60f113cf3 not found: ID does not exist"
Oct 31 19:46:18.948625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.948473    5551 scope.go:122] "RemoveContainer" containerID="1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97"
Oct 31 19:46:18.948764 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:46:18.948733    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97\": container with ID starting with 1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97 not found: ID does not exist" containerID="1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97"
Oct 31 19:46:18.948764 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.948750    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97"} err="failed to get container status \"1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97\": rpc error: code = NotFound desc = could not find container \"1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97\": container with ID starting with 1674ce113cf4799ef5a236a1b1f7d519076740c5ba886f510c14d03d69b3ac97 not found: ID does not exist"
Oct 31 19:46:18.977342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.977230    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:18.977342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.977269    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:18.977342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.977285    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmz7c\" (UniqueName: \"kubernetes.io/projected/611474c7-427f-438e-83ed-257b09db4ff1-kube-api-access-mmz7c\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:18.977342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.977301    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/611474c7-427f-438e-83ed-257b09db4ff1-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:18.977342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:18.977314    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/611474c7-427f-438e-83ed-257b09db4ff1-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:19.087823 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.087796    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:19.095814 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.095788    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"]
Oct 31 19:46:19.104038 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.104012    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f0e4e-openshift-default-696ffbf775-k4dt7"]
Oct 31 19:46:19.178740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.178643    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781a1a0e-264a-49c6-abcb-4255d46a8e5b-kube-api-access-r9xgt\" (UniqueName: \"kubernetes.io/projected/781a1a0e-264a-49c6-abcb-4255d46a8e5b-kube-api-access-r9xgt\") pod \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") "
Oct 31 19:46:19.178740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.178734    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-openshift-ca\") pod \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") "
Oct 31 19:46:19.178949 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.178832    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-xks-ca\") pod \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\" (UID: \"781a1a0e-264a-49c6-abcb-4255d46a8e5b\") "
Oct 31 19:46:19.179135 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.179103    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-openshift-ca" pod "781a1a0e-264a-49c6-abcb-4255d46a8e5b" (UID: "781a1a0e-264a-49c6-abcb-4255d46a8e5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:19.181096 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.181067    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-xks-ca" pod "781a1a0e-264a-49c6-abcb-4255d46a8e5b" (UID: "781a1a0e-264a-49c6-abcb-4255d46a8e5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:19.181207 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.181145    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781a1a0e-264a-49c6-abcb-4255d46a8e5b-kube-api-access-r9xgt" pod "781a1a0e-264a-49c6-abcb-4255d46a8e5b" (UID: "781a1a0e-264a-49c6-abcb-4255d46a8e5b"). InnerVolumeSpecName "kube-api-access-r9xgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:19.280534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.280497    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:19.280534 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.280534    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9xgt\" (UniqueName: \"kubernetes.io/projected/781a1a0e-264a-49c6-abcb-4255d46a8e5b-kube-api-access-r9xgt\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:19.280748 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.280550    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781a1a0e-264a-49c6-abcb-4255d46a8e5b-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:19.934712 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.934680    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"
Oct 31 19:46:19.935112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.934676    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr" event={"ID":"781a1a0e-264a-49c6-abcb-4255d46a8e5b","Type":"ContainerDied","Data":"67dded2e69a1a17bc014eb456edb821dfcf8b79b4c92bb44511d090d18e4ab78"}
Oct 31 19:46:19.935112 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:19.934817    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67dded2e69a1a17bc014eb456edb821dfcf8b79b4c92bb44511d090d18e4ab78"
Oct 31 19:46:20.908035 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:20.908001    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="611474c7-427f-438e-83ed-257b09db4ff1" path="/var/lib/kubelet/pods/611474c7-427f-438e-83ed-257b09db4ff1/volumes"
Oct 31 19:46:37.008586 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.008550    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"]
Oct 31 19:46:37.009057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009002    5551 cpu_manager.go:414] "containerMap: removing container" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy"
Oct 31 19:46:37.009057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009013    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy"
Oct 31 19:46:37.009057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009022    5551 cpu_manager.go:414] "containerMap: removing container" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy"
Oct 31 19:46:37.009057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009026    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy"
Oct 31 19:46:37.009057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009037    5551 cpu_manager.go:414] "containerMap: removing container" podUID="781a1a0e-264a-49c6-abcb-4255d46a8e5b" containerName="revoke-keys"
Oct 31 19:46:37.009057 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009043    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781a1a0e-264a-49c6-abcb-4255d46a8e5b" containerName="revoke-keys"
Oct 31 19:46:37.009259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009106    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe4c64f-6891-4be8-9021-33a63ef9d722" containerName="cleanup"
Oct 31 19:46:37.009259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009116    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy"
Oct 31 19:46:37.009259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009120    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="611474c7-427f-438e-83ed-257b09db4ff1" containerName="istio-proxy"
Oct 31 19:46:37.009259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.009126    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="781a1a0e-264a-49c6-abcb-4255d46a8e5b" containerName="revoke-keys"
Oct 31 19:46:37.013512 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.013493    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.060385 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.060355    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54637ff8-6fa5-4842-9676-742415732c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.060599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.060506    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54637ff8-6fa5-4842-9676-742415732c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.060599 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.060566    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmnlh\" (UniqueName: \"kubernetes.io/projected/54637ff8-6fa5-4842-9676-742415732c78-kube-api-access-gmnlh\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.065839 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.065806    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:46:37.115334 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.115306    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:46:37.115334 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.115320    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:46:37.115581 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.115373    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:46:37.118680 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.118658    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"]
Oct 31 19:46:37.161296 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.161274    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54637ff8-6fa5-4842-9676-742415732c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.161421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.161359    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54637ff8-6fa5-4842-9676-742415732c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.161421 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.161412    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmnlh\" (UniqueName: \"kubernetes.io/projected/54637ff8-6fa5-4842-9676-742415732c78-kube-api-access-gmnlh\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.162036 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.162018    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54637ff8-6fa5-4842-9676-742415732c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.163919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.163901    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54637ff8-6fa5-4842-9676-742415732c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.281022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.280947    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:46:37.292733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.292703    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmnlh\" (UniqueName: \"kubernetes.io/projected/54637ff8-6fa5-4842-9676-742415732c78-kube-api-access-gmnlh\") pod \"maas-api-revoke-sub-4536ae30-4f78g\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.330079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.330052    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:37.715186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:37.715162    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"]
Oct 31 19:46:37.718498 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:46:37.718448    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54637ff8_6fa5_4842_9676_742415732c78.slice/crio-95afdd0443e88f75ff33f8b9ab5e5774eb6d81406612ed711e93bc6b05709ea5 WatchSource:0}: Error finding container 95afdd0443e88f75ff33f8b9ab5e5774eb6d81406612ed711e93bc6b05709ea5: Status 404 returned error can't find the container with id 95afdd0443e88f75ff33f8b9ab5e5774eb6d81406612ed711e93bc6b05709ea5
Oct 31 19:46:38.044131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:38.044046    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g" event={"ID":"54637ff8-6fa5-4842-9676-742415732c78","Type":"ContainerStarted","Data":"ce4ce97410510d5f30d8d5a7e5f30cad7b4ca655a721aad4e1616a5c34dc7fe3"}
Oct 31 19:46:38.044131 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:38.044099    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g" event={"ID":"54637ff8-6fa5-4842-9676-742415732c78","Type":"ContainerStarted","Data":"95afdd0443e88f75ff33f8b9ab5e5774eb6d81406612ed711e93bc6b05709ea5"}
Oct 31 19:46:38.166179 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:38.166130    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g" podStartSLOduration=2.166115127 podStartE2EDuration="2.166115127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:38.165937655 +0000 UTC m=+5101.397175946" watchObservedRunningTime="2026-10-31 19:46:38.166115127 +0000 UTC m=+5101.397353416"
Oct 31 19:46:39.051024 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:39.050981    5551 generic.go:356] "Generic (PLEG): container finished" podID="54637ff8-6fa5-4842-9676-742415732c78" containerID="ce4ce97410510d5f30d8d5a7e5f30cad7b4ca655a721aad4e1616a5c34dc7fe3" exitCode=0
Oct 31 19:46:39.051441 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:39.051054    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g" event={"ID":"54637ff8-6fa5-4842-9676-742415732c78","Type":"ContainerDied","Data":"ce4ce97410510d5f30d8d5a7e5f30cad7b4ca655a721aad4e1616a5c34dc7fe3"}
Oct 31 19:46:40.210325 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.210299    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:40.295394 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.295366    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54637ff8-6fa5-4842-9676-742415732c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54637ff8-6fa5-4842-9676-742415732c78-maas-api-openshift-ca\") pod \"54637ff8-6fa5-4842-9676-742415732c78\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") "
Oct 31 19:46:40.295645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.295404    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54637ff8-6fa5-4842-9676-742415732c78-kube-api-access-gmnlh\" (UniqueName: \"kubernetes.io/projected/54637ff8-6fa5-4842-9676-742415732c78-kube-api-access-gmnlh\") pod \"54637ff8-6fa5-4842-9676-742415732c78\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") "
Oct 31 19:46:40.295645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.295531    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54637ff8-6fa5-4842-9676-742415732c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54637ff8-6fa5-4842-9676-742415732c78-maas-api-xks-ca\") pod \"54637ff8-6fa5-4842-9676-742415732c78\" (UID: \"54637ff8-6fa5-4842-9676-742415732c78\") "
Oct 31 19:46:40.295828 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.295805    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54637ff8-6fa5-4842-9676-742415732c78-maas-api-openshift-ca" pod "54637ff8-6fa5-4842-9676-742415732c78" (UID: "54637ff8-6fa5-4842-9676-742415732c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:40.295962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.295945    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54637ff8-6fa5-4842-9676-742415732c78-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:40.297708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.297684    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54637ff8-6fa5-4842-9676-742415732c78-maas-api-xks-ca" pod "54637ff8-6fa5-4842-9676-742415732c78" (UID: "54637ff8-6fa5-4842-9676-742415732c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:40.297834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.297816    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54637ff8-6fa5-4842-9676-742415732c78-kube-api-access-gmnlh" pod "54637ff8-6fa5-4842-9676-742415732c78" (UID: "54637ff8-6fa5-4842-9676-742415732c78"). InnerVolumeSpecName "kube-api-access-gmnlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:40.397516 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.397452    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmnlh\" (UniqueName: \"kubernetes.io/projected/54637ff8-6fa5-4842-9676-742415732c78-kube-api-access-gmnlh\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:40.397684 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.397520    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54637ff8-6fa5-4842-9676-742415732c78-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:40.711355 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.711276    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:46:40.722596 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:40.722560    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:46:41.064161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:41.064071    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g" event={"ID":"54637ff8-6fa5-4842-9676-742415732c78","Type":"ContainerDied","Data":"95afdd0443e88f75ff33f8b9ab5e5774eb6d81406612ed711e93bc6b05709ea5"}
Oct 31 19:46:41.064161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:41.064133    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95afdd0443e88f75ff33f8b9ab5e5774eb6d81406612ed711e93bc6b05709ea5"
Oct 31 19:46:41.064161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:41.064087    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"
Oct 31 19:46:46.604146 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:46:46.604110    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:46:59.143716 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:59.143681    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"]
Oct 31 19:46:59.144227 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:46:59.143945    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy" containerID="cri-o://a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5" gracePeriod=30
Oct 31 19:47:00.045429 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:00.045375    5551 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:47:00.045653 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:00.045487    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:47:00.282802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:00.282760    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d67dfb554-c2w6p_e22ad853-ddd2-4669-8df9-3c3f67f659f2/authorino/0.log"
Oct 31 19:47:00.706625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:00.706597    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-sbwg7_14e622e5-332a-4304-a9ec-d0feaa7c4002/manager/0.log"
Oct 31 19:47:00.950223 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:00.950183    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zkmz5_99d1f84b-a604-46c8-83be-3f53ab8c959c/manager/0.log"
Oct 31 19:47:01.583290 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:01.583253    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-q2zdr_e8c99ace-8a40-41aa-9615-809def49b98f/manager/0.log"
Oct 31 19:47:02.413055 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:02.413025    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hd4jn_2ff429fc-0746-4d6a-aff4-088caec1919e/manager/1.log"
Oct 31 19:47:03.120985 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:03.120950    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hd4jn_2ff429fc-0746-4d6a-aff4-088caec1919e/manager/0.log"
Oct 31 19:47:04.331362 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.331330    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sr4rh_58ef3323-2763-4091-8d61-f4600a97ad9b/manager/0.log"
Oct 31 19:47:04.398676 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.398654    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 19:47:04.449650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.449588    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-token\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.449650 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.449628    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-envoy\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.449837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.449690    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-certs\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.449837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.449734    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-podinfo\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.449837 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.449825    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-socket\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.450008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.449903    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istiod-ca-cert\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.450008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.449943    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-kube-api-access-rcwsb\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-kube-api-access-rcwsb\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.450008 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.449999    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-data\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.450182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.450037    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-credential-socket\") pod \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\" (UID: \"9a0e9132-e0de-4ad8-bc9a-0e621973e643\") "
Oct 31 19:47:04.450182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.450059    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-certs" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.450182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.450107    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-socket" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.450408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.450390    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.450516 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.450365    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-data" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.450591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.450515    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istiod-ca-cert" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:04.450591 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.450412    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.450710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.450649    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-credential-socket" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.451922 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.451895    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-token" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:04.452017 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.451933    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-envoy" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.452569 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.452552    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-podinfo" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:47:04.452640 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.452606    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-kube-api-access-rcwsb" pod "9a0e9132-e0de-4ad8-bc9a-0e621973e643" (UID: "9a0e9132-e0de-4ad8-bc9a-0e621973e643"). InnerVolumeSpecName "kube-api-access-rcwsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:04.551636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.551602    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.551636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.551636    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcwsb\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-kube-api-access-rcwsb\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.551786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.551652    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.551786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.551660    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.551786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.551669    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.551786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.551677    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.551786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:04.551685    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9132-e0de-4ad8-bc9a-0e621973e643-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:05.205490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.205379    5551 generic.go:356] "Generic (PLEG): container finished" podID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerID="a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5" exitCode=0
Oct 31 19:47:05.205490 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.205442    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"
Oct 31 19:47:05.205688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.205437    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" event={"ID":"9a0e9132-e0de-4ad8-bc9a-0e621973e643","Type":"ContainerDied","Data":"a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5"}
Oct 31 19:47:05.205688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.205571    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j" event={"ID":"9a0e9132-e0de-4ad8-bc9a-0e621973e643","Type":"ContainerDied","Data":"e9cccab912d1b75149120a2f3bd9ace24b7138d610d06fb34ff6b7a865cf96a7"}
Oct 31 19:47:05.205688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.205593    5551 scope.go:122] "RemoveContainer" containerID="a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5"
Oct 31 19:47:05.215615 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.215594    5551 scope.go:122] "RemoveContainer" containerID="ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b"
Oct 31 19:47:05.224964 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.224942    5551 scope.go:122] "RemoveContainer" containerID="a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5"
Oct 31 19:47:05.225312 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:47:05.225286    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5\": container with ID starting with a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5 not found: ID does not exist" containerID="a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5"
Oct 31 19:47:05.225441 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.225325    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5"} err="failed to get container status \"a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5\": rpc error: code = NotFound desc = could not find container \"a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5\": container with ID starting with a485ef16e959974a831b52034c15c137cf224a516c38d5149585a3502c4f7bf5 not found: ID does not exist"
Oct 31 19:47:05.225441 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.225350    5551 scope.go:122] "RemoveContainer" containerID="ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b"
Oct 31 19:47:05.225683 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:47:05.225664    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b\": container with ID starting with ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b not found: ID does not exist" containerID="ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b"
Oct 31 19:47:05.225726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.225691    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b"} err="failed to get container status \"ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b\": rpc error: code = NotFound desc = could not find container \"ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b\": container with ID starting with ffd0a6df158f1d485441309ecf0958583a44c10670966f9901a7f38e1efa9a3b not found: ID does not exist"
Oct 31 19:47:05.369671 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.369634    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"]
Oct 31 19:47:05.419658 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:05.419628    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-wlj2j"]
Oct 31 19:47:06.916041 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:06.915996    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" path="/var/lib/kubelet/pods/9a0e9132-e0de-4ad8-bc9a-0e621973e643/volumes"
Oct 31 19:47:16.604296 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:47:16.604204    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:47:20.309586 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.309553    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"]
Oct 31 19:47:20.310226 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.310207    5551 cpu_manager.go:414] "containerMap: removing container" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy"
Oct 31 19:47:20.310275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.310228    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy"
Oct 31 19:47:20.310275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.310246    5551 cpu_manager.go:414] "containerMap: removing container" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy"
Oct 31 19:47:20.310275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.310252    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy"
Oct 31 19:47:20.310275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.310266    5551 cpu_manager.go:414] "containerMap: removing container" podUID="54637ff8-6fa5-4842-9676-742415732c78" containerName="revoke-keys"
Oct 31 19:47:20.310275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.310271    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54637ff8-6fa5-4842-9676-742415732c78" containerName="revoke-keys"
Oct 31 19:47:20.310444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.310340    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy"
Oct 31 19:47:20.310444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.310352    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="54637ff8-6fa5-4842-9676-742415732c78" containerName="revoke-keys"
Oct 31 19:47:20.313537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.313521    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.365420 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.365381    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:47:20.414818 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.414791    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-934ace10-serving-cert\""
Oct 31 19:47:20.415042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.414802    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 19:47:20.415042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.414852    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:47:20.415042 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.414894    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 19:47:20.416342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.416315    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ngz\" (UniqueName: \"kubernetes.io/projected/26fa603f-6563-4cb3-997f-d946f8ba33ed-kube-api-access-94ngz\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.416450 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.416390    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-metrics-certs\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.416526 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.416497    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fa603f-6563-4cb3-997f-d946f8ba33ed-openshift-service-ca\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.416588 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.416550    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.417331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.417310    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"]
Oct 31 19:47:20.517889 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.517858    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ngz\" (UniqueName: \"kubernetes.io/projected/26fa603f-6563-4cb3-997f-d946f8ba33ed-kube-api-access-94ngz\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.518079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.517911    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-metrics-certs\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.518079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.517930    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fa603f-6563-4cb3-997f-d946f8ba33ed-openshift-service-ca\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.518079 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.517953    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.518200 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:47:20.518084    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-serving-cert: secret "maas-api-e2e-ait-934ace10-serving-cert" not found
Oct 31 19:47:20.518200 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:47:20.518148    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls podName:26fa603f-6563-4cb3-997f-d946f8ba33ed nodeName:}" failed. No retries permitted until 2026-10-31 19:47:21.018129219 +0000 UTC m=+5144.249367486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls") pod "maas-api-e2e-ait-934ace10-5b8648554b-sdnsh" (UID: "26fa603f-6563-4cb3-997f-d946f8ba33ed") : secret "maas-api-e2e-ait-934ace10-serving-cert" not found
Oct 31 19:47:20.518721 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.518699    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fa603f-6563-4cb3-997f-d946f8ba33ed-openshift-service-ca\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.520568 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.520543    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-metrics-certs\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:20.577851 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:20.577773    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ngz\" (UniqueName: \"kubernetes.io/projected/26fa603f-6563-4cb3-997f-d946f8ba33ed-kube-api-access-94ngz\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:21.023295 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:21.023261    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:21.025792 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:21.025767    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls\") pod \"maas-api-e2e-ait-934ace10-5b8648554b-sdnsh\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:21.226099 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:21.226069    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:21.603734 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:21.603677    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"]
Oct 31 19:47:21.606130 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:47:21.606099    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fa603f_6563_4cb3_997f_d946f8ba33ed.slice/crio-5285b10414b27192f84de4450073886b9f0b92b9ccda6c3e34aac5d6ead1ad08 WatchSource:0}: Error finding container 5285b10414b27192f84de4450073886b9f0b92b9ccda6c3e34aac5d6ead1ad08: Status 404 returned error can't find the container with id 5285b10414b27192f84de4450073886b9f0b92b9ccda6c3e34aac5d6ead1ad08
Oct 31 19:47:21.607483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:21.607447    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:47:22.307292 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:22.307195    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh" event={"ID":"26fa603f-6563-4cb3-997f-d946f8ba33ed","Type":"ContainerStarted","Data":"5285b10414b27192f84de4450073886b9f0b92b9ccda6c3e34aac5d6ead1ad08"}
Oct 31 19:47:23.314648 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:23.314581    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh" event={"ID":"26fa603f-6563-4cb3-997f-d946f8ba33ed","Type":"ContainerStarted","Data":"952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796"}
Oct 31 19:47:23.315123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:23.314701    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:23.430505 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:23.430433    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh" podStartSLOduration=2.482106264 podStartE2EDuration="3.430419378s" totalImagesPullingTime="948.313114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:23.428584735 +0000 UTC m=+5146.659823025" watchObservedRunningTime="2026-10-31 19:47:23.430419378 +0000 UTC m=+5146.661657667"
Oct 31 19:47:29.326036 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:47:29.326001    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:47:46.605434 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:47:46.605392    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:48:08.062740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.062711    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"]
Oct 31 19:48:08.063313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.062933    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh" podUID="26fa603f-6563-4cb3-997f-d946f8ba33ed" containerName="maas-api" containerID="cri-o://952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796" gracePeriod=30
Oct 31 19:48:08.322346 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.322292    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:48:08.390879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.390852    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-metrics-certs\") pod \"26fa603f-6563-4cb3-997f-d946f8ba33ed\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") "
Oct 31 19:48:08.391018 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.390884    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls\") pod \"26fa603f-6563-4cb3-997f-d946f8ba33ed\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") "
Oct 31 19:48:08.391066 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.391035    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fa603f-6563-4cb3-997f-d946f8ba33ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fa603f-6563-4cb3-997f-d946f8ba33ed-openshift-service-ca\") pod \"26fa603f-6563-4cb3-997f-d946f8ba33ed\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") "
Oct 31 19:48:08.391066 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.391056    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fa603f-6563-4cb3-997f-d946f8ba33ed-kube-api-access-94ngz\" (UniqueName: \"kubernetes.io/projected/26fa603f-6563-4cb3-997f-d946f8ba33ed-kube-api-access-94ngz\") pod \"26fa603f-6563-4cb3-997f-d946f8ba33ed\" (UID: \"26fa603f-6563-4cb3-997f-d946f8ba33ed\") "
Oct 31 19:48:08.391397 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.391370    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fa603f-6563-4cb3-997f-d946f8ba33ed-openshift-service-ca" pod "26fa603f-6563-4cb3-997f-d946f8ba33ed" (UID: "26fa603f-6563-4cb3-997f-d946f8ba33ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:08.393078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.393049    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls" pod "26fa603f-6563-4cb3-997f-d946f8ba33ed" (UID: "26fa603f-6563-4cb3-997f-d946f8ba33ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:08.393184 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.393069    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-metrics-certs" pod "26fa603f-6563-4cb3-997f-d946f8ba33ed" (UID: "26fa603f-6563-4cb3-997f-d946f8ba33ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:08.393238 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.393172    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fa603f-6563-4cb3-997f-d946f8ba33ed-kube-api-access-94ngz" pod "26fa603f-6563-4cb3-997f-d946f8ba33ed" (UID: "26fa603f-6563-4cb3-997f-d946f8ba33ed"). InnerVolumeSpecName "kube-api-access-94ngz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:08.492279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.492236    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:08.492279 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.492279    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fa603f-6563-4cb3-997f-d946f8ba33ed-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:08.492551 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.492296    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fa603f-6563-4cb3-997f-d946f8ba33ed-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:08.492551 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.492311    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94ngz\" (UniqueName: \"kubernetes.io/projected/26fa603f-6563-4cb3-997f-d946f8ba33ed-kube-api-access-94ngz\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:08.559177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.559145    5551 generic.go:356] "Generic (PLEG): container finished" podID="26fa603f-6563-4cb3-997f-d946f8ba33ed" containerID="952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796" exitCode=0
Oct 31 19:48:08.559370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.559196    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"
Oct 31 19:48:08.559370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.559230    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh" event={"ID":"26fa603f-6563-4cb3-997f-d946f8ba33ed","Type":"ContainerDied","Data":"952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796"}
Oct 31 19:48:08.559370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.559288    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh" event={"ID":"26fa603f-6563-4cb3-997f-d946f8ba33ed","Type":"ContainerDied","Data":"5285b10414b27192f84de4450073886b9f0b92b9ccda6c3e34aac5d6ead1ad08"}
Oct 31 19:48:08.559370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.559312    5551 scope.go:122] "RemoveContainer" containerID="952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796"
Oct 31 19:48:08.573116 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.573090    5551 scope.go:122] "RemoveContainer" containerID="952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796"
Oct 31 19:48:08.573504 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:48:08.573482    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796\": container with ID starting with 952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796 not found: ID does not exist" containerID="952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796"
Oct 31 19:48:08.573595 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.573520    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796"} err="failed to get container status \"952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796\": rpc error: code = NotFound desc = could not find container \"952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796\": container with ID starting with 952077ce4ef521bea1e4101582d304af41c01cba8cd7894faa9f2c4466c8a796 not found: ID does not exist"
Oct 31 19:48:08.722674 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.722634    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"]
Oct 31 19:48:08.727912 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.727877    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-934ace10-5b8648554b-sdnsh"]
Oct 31 19:48:08.907678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:08.907650    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fa603f-6563-4cb3-997f-d946f8ba33ed" path="/var/lib/kubelet/pods/26fa603f-6563-4cb3-997f-d946f8ba33ed/volumes"
Oct 31 19:48:16.610746 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:48:16.610704    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:48:21.372093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:21.372050    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"]
Oct 31 19:48:21.372616 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:21.372312    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy" containerID="cri-o://dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440" gracePeriod=30
Oct 31 19:48:26.625483 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.625438    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:48:26.669177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.669090    5551 generic.go:356] "Generic (PLEG): container finished" podID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerID="dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440" exitCode=0
Oct 31 19:48:26.669177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.669175    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" event={"ID":"93479335-6d59-4b1c-b4d4-0a9481688bc3","Type":"ContainerDied","Data":"dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440"}
Oct 31 19:48:26.669364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.669203    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"
Oct 31 19:48:26.669364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.669239    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt" event={"ID":"93479335-6d59-4b1c-b4d4-0a9481688bc3","Type":"ContainerDied","Data":"3b5ceddebd6da3f69b15802db5fc0133f714b86662cd3b5d5bc371dc26acb407"}
Oct 31 19:48:26.669364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.669260    5551 scope.go:122] "RemoveContainer" containerID="dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440"
Oct 31 19:48:26.678621 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.678604    5551 scope.go:122] "RemoveContainer" containerID="dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440"
Oct 31 19:48:26.678881 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:48:26.678862    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440\": container with ID starting with dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440 not found: ID does not exist" containerID="dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440"
Oct 31 19:48:26.678964 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.678890    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440"} err="failed to get container status \"dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440\": rpc error: code = NotFound desc = could not find container \"dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440\": container with ID starting with dd772abefb1d18861981d3510b62bb34ccc4992cc09f0364dd9ab7589064f440 not found: ID does not exist"
Oct 31 19:48:26.780404 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780379    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-credential-socket\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.780519 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780435    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-envoy\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.780519 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780488    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-podinfo\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.780619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780525    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-data\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.780619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780551    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-certs\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.780619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780572    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93479335-6d59-4b1c-b4d4-0a9481688bc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93479335-6d59-4b1c-b4d4-0a9481688bc3-istiod-ca-cert\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.780619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780606    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-kube-api-access-jq22s\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-kube-api-access-jq22s\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.780833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780656    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-socket\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.780833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780668    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-credential-socket" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:26.780833 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780687    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-token\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-token\") pod \"93479335-6d59-4b1c-b4d4-0a9481688bc3\" (UID: \"93479335-6d59-4b1c-b4d4-0a9481688bc3\") "
Oct 31 19:48:26.781007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780862    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-certs" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:26.781007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780900    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-socket" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:26.781007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780948    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-data" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:26.781007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.780986    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93479335-6d59-4b1c-b4d4-0a9481688bc3-istiod-ca-cert" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:26.781236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.781070    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:26.781236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.781097    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:26.781236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.781115    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:26.781236 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.781129    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:26.782800 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.782774    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-envoy" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:26.782960 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.782927    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-kube-api-access-jq22s" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "kube-api-access-jq22s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:26.783068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.782987    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-podinfo" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:26.783068 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.783024    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-token" pod "93479335-6d59-4b1c-b4d4-0a9481688bc3" (UID: "93479335-6d59-4b1c-b4d4-0a9481688bc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:26.881959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.881928    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:26.881959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.881955    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:26.881959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.881966    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93479335-6d59-4b1c-b4d4-0a9481688bc3-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:26.882181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.881974    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq22s\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-kube-api-access-jq22s\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:26.882181 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:26.881985    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93479335-6d59-4b1c-b4d4-0a9481688bc3-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.084848 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:27.084753    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"]
Oct 31 19:48:27.091321 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:27.091287    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hzvrt"]
Oct 31 19:48:28.907329 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:48:28.907295    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" path="/var/lib/kubelet/pods/93479335-6d59-4b1c-b4d4-0a9481688bc3/volumes"
Oct 31 19:48:46.603867 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:48:46.603782    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:49:16.605104 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:49:16.605068    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:49:46.603553 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:49:46.603518    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:49:52.094022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.093989    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"]
Oct 31 19:49:52.094559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.094384    5551 cpu_manager.go:414] "containerMap: removing container" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy"
Oct 31 19:49:52.094559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.094396    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy"
Oct 31 19:49:52.094559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.094407    5551 cpu_manager.go:414] "containerMap: removing container" podUID="26fa603f-6563-4cb3-997f-d946f8ba33ed" containerName="maas-api"
Oct 31 19:49:52.094559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.094412    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fa603f-6563-4cb3-997f-d946f8ba33ed" containerName="maas-api"
Oct 31 19:49:52.094559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.094518    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="93479335-6d59-4b1c-b4d4-0a9481688bc3" containerName="istio-proxy"
Oct 31 19:49:52.094559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.094526    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fa603f-6563-4cb3-997f-d946f8ba33ed" containerName="maas-api"
Oct 31 19:49:52.094559 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.094534    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0e9132-e0de-4ad8-bc9a-0e621973e643" containerName="istio-proxy"
Oct 31 19:49:52.097600 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.097579    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.192868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.192840    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"]
Oct 31 19:49:52.210167 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210141    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6cba1a-6e61-4893-a34e-0690f844e329-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.210324 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210198    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.210384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210322    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.210446 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210427    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.210518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210481    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.210518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210510    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96nf\" (UniqueName: \"kubernetes.io/projected/5d6cba1a-6e61-4893-a34e-0690f844e329-kube-api-access-q96nf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.210637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210564    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.210637 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210610    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.210723 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.210683    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.311877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.311843    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96nf\" (UniqueName: \"kubernetes.io/projected/5d6cba1a-6e61-4893-a34e-0690f844e329-kube-api-access-q96nf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.311877 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.311878    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.311910    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312027    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312115    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6cba1a-6e61-4893-a34e-0690f844e329-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312169    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312197    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312238    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312315 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312294    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312318    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312401    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312561 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312545    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312708    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.312822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.312806    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6cba1a-6e61-4893-a34e-0690f844e329-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.314635 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.314613    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.314741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.314724    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.367928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.367853    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6cba1a-6e61-4893-a34e-0690f844e329-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.367928 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.367909    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96nf\" (UniqueName: \"kubernetes.io/projected/5d6cba1a-6e61-4893-a34e-0690f844e329-kube-api-access-q96nf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mjzjn\" (UID: \"5d6cba1a-6e61-4893-a34e-0690f844e329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.410773 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.410744    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:52.839128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.839056    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"]
Oct 31 19:49:52.841420 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:49:52.841384    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6cba1a_6e61_4893_a34e_0690f844e329.slice/crio-22acee5354bbbe501fe4a0e45594474bc6d1804ea3b29f62b53c5c1bb7f7ca37 WatchSource:0}: Error finding container 22acee5354bbbe501fe4a0e45594474bc6d1804ea3b29f62b53c5c1bb7f7ca37: Status 404 returned error can't find the container with id 22acee5354bbbe501fe4a0e45594474bc6d1804ea3b29f62b53c5c1bb7f7ca37
Oct 31 19:49:52.843619 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.843587    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:49:52.843708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.843650    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:49:52.843708 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:52.843682    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:49:53.156717 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:53.156678    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" event={"ID":"5d6cba1a-6e61-4893-a34e-0690f844e329","Type":"ContainerStarted","Data":"e237cbe9a2cc928a468f34e4f1ba218147d5883180e22bc3ae8da2edc6e34603"}
Oct 31 19:49:53.157311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:53.157278    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" event={"ID":"5d6cba1a-6e61-4893-a34e-0690f844e329","Type":"ContainerStarted","Data":"22acee5354bbbe501fe4a0e45594474bc6d1804ea3b29f62b53c5c1bb7f7ca37"}
Oct 31 19:49:53.411161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:53.411061    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:49:53.412685 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:53.412654    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 19:49:53.412832 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:53.412713    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 19:49:54.411403 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:54.411367    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 19:49:54.411852 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:54.411432    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 19:49:55.411439 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:55.411402    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 19:49:55.411868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:55.411483    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 19:49:56.411736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:56.411692    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 19:49:56.412215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:56.411766    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 19:49:57.411440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:57.411400    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 19:49:57.411706 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:57.411486    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 19:49:58.411261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:58.411224    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 19:49:58.411703 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:58.411286    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 19:49:59.411407 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:59.411356    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 19:49:59.411965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:49:59.411434    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 19:50:01.411177 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:01.411132    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:50:01.411741 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:01.411199    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:02.452656 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:02.452604    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:50:02.453191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:02.452674    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:03.453476 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:03.453413    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:50:03.454017 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:03.453494    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:03.478728 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:03.478701    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:50:03.479123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:03.479098    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:50:03.479892 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:03.479874    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn"
Oct 31 19:50:03.624641 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:03.624591    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podStartSLOduration=11.62457856 podStartE2EDuration="11.62457856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:53.276265771 +0000 UTC m=+5296.507504061" watchObservedRunningTime="2026-10-31 19:50:03.62457856 +0000 UTC m=+5306.855816849"
Oct 31 19:50:16.604253 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:50:16.604164    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:50:22.338743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.338646    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"]
Oct 31 19:50:22.342519 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.342497    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.408496 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.399656    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"]
Oct 31 19:50:22.408496 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.404794    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.443872 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.443840    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j787d\" (UniqueName: \"kubernetes.io/projected/e64f2c2f-b471-4e6e-a597-a11a9975a428-kube-api-access-j787d\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.444038 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.443877    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.444038 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.443911    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64f2c2f-b471-4e6e-a597-a11a9975a428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.444161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.444067    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.444161 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.444112    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.444231 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.444203    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.444284 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.444266    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.444371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.444356    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.444431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.444409    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.525484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.524718    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"]
Oct 31 19:50:22.545277 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545249    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.545277 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545280    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.545555 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545310    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.545555 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545376    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.545555 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545415    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.545555 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545489    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.545944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545623    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.545944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545659    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.545944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545719    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.545944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545768    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.545944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545822    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.545944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545890    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.545944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545898    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.545944 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545924    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j787d\" (UniqueName: \"kubernetes.io/projected/e64f2c2f-b471-4e6e-a597-a11a9975a428-kube-api-access-j787d\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.546398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.545977    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.546398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.546005    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.546398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.546046    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.546398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.546056    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjplk\" (UniqueName: \"kubernetes.io/projected/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-kube-api-access-cjplk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.546398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.546208    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64f2c2f-b471-4e6e-a597-a11a9975a428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.546398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.546258    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.546398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.546280    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.546751 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.546545    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.546950 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.546924    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64f2c2f-b471-4e6e-a597-a11a9975a428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.547987 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.547965    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.548611 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.548585    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.598567 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.579038    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"]
Oct 31 19:50:22.621627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.621111    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64f2c2f-b471-4e6e-a597-a11a9975a428-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.624541 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.624488    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j787d\" (UniqueName: \"kubernetes.io/projected/e64f2c2f-b471-4e6e-a597-a11a9975a428-kube-api-access-j787d\") pod \"maas-default-gateway-openshift-default-7f694df8bc-8z84d\" (UID: \"e64f2c2f-b471-4e6e-a597-a11a9975a428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.647713 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.647689    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.647816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.647718    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.647816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.647745    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.647816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.647761    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.647816 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.647783    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.647985 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.647907    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.647985 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.647945    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.648100 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.647993    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjplk\" (UniqueName: \"kubernetes.io/projected/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-kube-api-access-cjplk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.648100 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.648025    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.648213 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.648146    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.648272 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.648213    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.648400 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.648381    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.648634 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.648526    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.648634 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.648573    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.657479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.653682    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.657479 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.656964    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.666486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.665695    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:22.710447 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.710418    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.711066 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.711041    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjplk\" (UniqueName: \"kubernetes.io/projected/75e922ca-483e-4f40-b4c0-c47f5fa1a9cb-kube-api-access-cjplk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-b8w4r\" (UID: \"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:22.724498 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:22.724450    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:23.138542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.138509    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"]
Oct 31 19:50:23.142251 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:50:23.142209    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64f2c2f_b471_4e6e_a597_a11a9975a428.slice/crio-a8b62985893982245089200d18815de15f3430cd750b83348d767e5c0d528e68 WatchSource:0}: Error finding container a8b62985893982245089200d18815de15f3430cd750b83348d767e5c0d528e68: Status 404 returned error can't find the container with id a8b62985893982245089200d18815de15f3430cd750b83348d767e5c0d528e68
Oct 31 19:50:23.144548 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.144515    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:50:23.144628 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.144588    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:50:23.144666 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.144630    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:50:23.203964 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.203936    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"]
Oct 31 19:50:23.212509 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.212445    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:50:23.212799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.212753    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:50:23.213007 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.212984    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:50:23.403890 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.403800    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" event={"ID":"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb","Type":"ContainerStarted","Data":"12f7023ad7abf40f64b755d881890dc06ee2bb121117915daa81b4e913c88b28"}
Oct 31 19:50:23.407610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.407501    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" event={"ID":"75e922ca-483e-4f40-b4c0-c47f5fa1a9cb","Type":"ContainerStarted","Data":"0a817733d962efb4e4c078333bc7717608b5d81fca335b0b070227d0a9dcfd18"}
Oct 31 19:50:23.407610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.407541    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" event={"ID":"e64f2c2f-b471-4e6e-a597-a11a9975a428","Type":"ContainerStarted","Data":"46dc7c92f4209188f260a95a9d86080d06d3824c33f015e6c6b23d303b0be46a"}
Oct 31 19:50:23.407610 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.407565    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" event={"ID":"e64f2c2f-b471-4e6e-a597-a11a9975a428","Type":"ContainerStarted","Data":"a8b62985893982245089200d18815de15f3430cd750b83348d767e5c0d528e68"}
Oct 31 19:50:23.521381 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.521315    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podStartSLOduration=1.521295836 podStartE2EDuration="1.521295836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:23.519991956 +0000 UTC m=+5326.751230245" watchObservedRunningTime="2026-10-31 19:50:23.521295836 +0000 UTC m=+5326.752534128"
Oct 31 19:50:23.639180 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.639106    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podStartSLOduration=1.639086497 podStartE2EDuration="1.639086497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:23.635728336 +0000 UTC m=+5326.866966651" watchObservedRunningTime="2026-10-31 19:50:23.639086497 +0000 UTC m=+5326.870324788"
Oct 31 19:50:23.666422 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.666345    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:23.728485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:23.725417    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:24.676484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:24.674570    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:50:24.676484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:24.674635    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 31 19:50:24.727075 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:24.727041    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:50:24.727259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:24.727095    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 31 19:50:25.666872 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:25.666836    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:50:25.667170 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:25.667142    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 31 19:50:25.725348 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:25.725311    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:50:25.725794 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:25.725374    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 31 19:50:26.672485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:26.668815    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:50:26.672485 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:26.668878    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 31 19:50:26.725792 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:26.725751    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:50:26.726183 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:26.725811    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 31 19:50:27.668881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:27.668841    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:50:27.669234 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:27.669192    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 31 19:50:27.725384 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:27.725342    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:50:27.725598 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:27.725423    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 31 19:50:28.672484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:28.668565    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:50:28.672484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:28.668626    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 31 19:50:28.725197 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:28.725162    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:50:28.725366 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:28.725211    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 31 19:50:29.667629 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:29.667574    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:50:29.667850 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:29.667652    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 31 19:50:29.725699 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:29.725656    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:50:29.726125 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:29.725726    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 31 19:50:30.667401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:30.667362    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:50:30.667615 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:30.667424    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Oct 31 19:50:30.725733 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:30.725699    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:50:30.726110 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:30.725761    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused"
Oct 31 19:50:31.671043 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:31.671014    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:31.671503 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:31.671481    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:31.672291 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:31.672270    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d"
Oct 31 19:50:31.731606 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:31.731576    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:31.732078 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:31.731978    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:31.732857 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:31.732834    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r"
Oct 31 19:50:46.603380 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:50:46.603341    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:50:51.435881 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.435847    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"]
Oct 31 19:50:51.439535 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.439516    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.547669 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.547603    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-466e73b1-kserve-self-signed-certs\""
Oct 31 19:50:51.549258 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.549229    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"]
Oct 31 19:50:51.571735 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.571701    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lv87\" (UniqueName: \"kubernetes.io/projected/d76d33d0-994e-431c-895f-600b7da8141e-kube-api-access-2lv87\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.572011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.571975    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-home\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.572220 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.572191    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-model-cache\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.572443 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.572413    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76d33d0-994e-431c-895f-600b7da8141e-tls-certs\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.572617 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.572598    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-dshm\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.572752 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.572736    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-tmp-dir\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.703186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.703090    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lv87\" (UniqueName: \"kubernetes.io/projected/d76d33d0-994e-431c-895f-600b7da8141e-kube-api-access-2lv87\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.703439 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.703415    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-home\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.703663 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.703643    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-model-cache\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.703873 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.703854    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76d33d0-994e-431c-895f-600b7da8141e-tls-certs\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.704014 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.703996    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-dshm\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.704133 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.704119    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-tmp-dir\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.704736 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.704712    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-tmp-dir\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.722480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.705412    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-home\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.722919 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.722895    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-model-cache\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.739739 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.739716    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-dshm\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.741612 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.741555    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76d33d0-994e-431c-895f-600b7da8141e-tls-certs\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:51.775136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:51.775103    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lv87\" (UniqueName: \"kubernetes.io/projected/d76d33d0-994e-431c-895f-600b7da8141e-kube-api-access-2lv87\") pod \"e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") " pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:52.065786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:52.065684    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:50:52.466225 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:52.466195    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"]
Oct 31 19:50:52.468140 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:50:52.468113    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd76d33d0_994e_431c_895f_600b7da8141e.slice/crio-27b0c3423cedb3a25f4290f0b3e2fc688b2fffeb044433aab62df2cb9452156b WatchSource:0}: Error finding container 27b0c3423cedb3a25f4290f0b3e2fc688b2fffeb044433aab62df2cb9452156b: Status 404 returned error can't find the container with id 27b0c3423cedb3a25f4290f0b3e2fc688b2fffeb044433aab62df2cb9452156b
Oct 31 19:50:52.678371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:52.678336    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2" event={"ID":"d76d33d0-994e-431c-895f-600b7da8141e","Type":"ContainerStarted","Data":"4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7"}
Oct 31 19:50:52.678371 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:52.678376    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2" event={"ID":"d76d33d0-994e-431c-895f-600b7da8141e","Type":"ContainerStarted","Data":"27b0c3423cedb3a25f4290f0b3e2fc688b2fffeb044433aab62df2cb9452156b"}
Oct 31 19:50:52.803047 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:52.802931    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2" podStartSLOduration=1.8029163320000001 podStartE2EDuration="1.802916332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:52.801451446 +0000 UTC m=+5356.032689737" watchObservedRunningTime="2026-10-31 19:50:52.802916332 +0000 UTC m=+5356.034154622"
Oct 31 19:50:53.067070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:50:53.066980    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:51:02.066809 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:02.066773    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:51:02.079367 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:02.079336    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:51:02.781259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:02.781231    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:51:16.604519 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:51:16.604480    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:51:20.047331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:20.047289    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"]
Oct 31 19:51:20.096518 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:20.096485    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3a852-w88mr"]
Oct 31 19:51:20.907438 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:20.907406    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781a1a0e-264a-49c6-abcb-4255d46a8e5b" path="/var/lib/kubelet/pods/781a1a0e-264a-49c6-abcb-4255d46a8e5b/volumes"
Oct 31 19:51:40.762823 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:40.762793    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:51:40.767431 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:40.767406    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:51:41.055695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:41.055564    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"]
Oct 31 19:51:41.105070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:41.105033    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4536ae30-4f78g"]
Oct 31 19:51:42.907853 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:42.907777    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54637ff8-6fa5-4842-9676-742415732c78" path="/var/lib/kubelet/pods/54637ff8-6fa5-4842-9676-742415732c78/volumes"
Oct 31 19:51:46.206499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.206448    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"]
Oct 31 19:51:46.211399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.211381    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.314256 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.314207    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:51:46.314256 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.314223    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:51:46.314256 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.314226    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:51:46.314256 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.314254    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:51:46.316574 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.316553    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"]
Oct 31 19:51:46.374070 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.374017    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.374793 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.374755    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.374966 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.374880    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9cp\" (UniqueName: \"kubernetes.io/projected/dd1aa024-ddd0-46da-8416-b1a69301c1a0-kube-api-access-2t9cp\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.476593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.476506    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.476593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.476586    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.476846 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.476630    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9cp\" (UniqueName: \"kubernetes.io/projected/dd1aa024-ddd0-46da-8416-b1a69301c1a0-kube-api-access-2t9cp\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.477307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.477278    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.479049 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.479015    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.533536 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.533505    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9cp\" (UniqueName: \"kubernetes.io/projected/dd1aa024-ddd0-46da-8416-b1a69301c1a0-kube-api-access-2t9cp\") pod \"maas-api-revoke-sub-e6c26f0e-cjgwp\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:46.603700 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:51:46.603662    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:51:46.824361 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:46.824286    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:47.201480 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:47.201442    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"]
Oct 31 19:51:47.203438 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:51:47.203412    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1aa024_ddd0_46da_8416_b1a69301c1a0.slice/crio-0d77ba5565bb77f101cb0057f3902a396538f0055f8a43a4ab193985933e7cb2 WatchSource:0}: Error finding container 0d77ba5565bb77f101cb0057f3902a396538f0055f8a43a4ab193985933e7cb2: Status 404 returned error can't find the container with id 0d77ba5565bb77f101cb0057f3902a396538f0055f8a43a4ab193985933e7cb2
Oct 31 19:51:48.094612 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:48.094580    5551 generic.go:356] "Generic (PLEG): container finished" podID="dd1aa024-ddd0-46da-8416-b1a69301c1a0" containerID="39748b6af24ef48e7b837a77533ae4acbb13bd1e423d03cdb1e3d5309e8f4902" exitCode=0
Oct 31 19:51:48.094967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:48.094617    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp" event={"ID":"dd1aa024-ddd0-46da-8416-b1a69301c1a0","Type":"ContainerDied","Data":"39748b6af24ef48e7b837a77533ae4acbb13bd1e423d03cdb1e3d5309e8f4902"}
Oct 31 19:51:48.094967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:48.094696    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp" event={"ID":"dd1aa024-ddd0-46da-8416-b1a69301c1a0","Type":"ContainerStarted","Data":"0d77ba5565bb77f101cb0057f3902a396538f0055f8a43a4ab193985933e7cb2"}
Oct 31 19:51:49.234029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.234002    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:49.406645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.406614    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1aa024-ddd0-46da-8416-b1a69301c1a0-kube-api-access-2t9cp\" (UniqueName: \"kubernetes.io/projected/dd1aa024-ddd0-46da-8416-b1a69301c1a0-kube-api-access-2t9cp\") pod \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") "
Oct 31 19:51:49.406815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.406670    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-xks-ca\") pod \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") "
Oct 31 19:51:49.406815 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.406711    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-openshift-ca\") pod \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\" (UID: \"dd1aa024-ddd0-46da-8416-b1a69301c1a0\") "
Oct 31 19:51:49.407143 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.407118    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-openshift-ca" pod "dd1aa024-ddd0-46da-8416-b1a69301c1a0" (UID: "dd1aa024-ddd0-46da-8416-b1a69301c1a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:49.408858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.408829    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-xks-ca" pod "dd1aa024-ddd0-46da-8416-b1a69301c1a0" (UID: "dd1aa024-ddd0-46da-8416-b1a69301c1a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:49.408948 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.408889    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1aa024-ddd0-46da-8416-b1a69301c1a0-kube-api-access-2t9cp" pod "dd1aa024-ddd0-46da-8416-b1a69301c1a0" (UID: "dd1aa024-ddd0-46da-8416-b1a69301c1a0"). InnerVolumeSpecName "kube-api-access-2t9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:49.507846 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.507818    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:51:49.507846 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.507845    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9cp\" (UniqueName: \"kubernetes.io/projected/dd1aa024-ddd0-46da-8416-b1a69301c1a0-kube-api-access-2t9cp\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:51:49.508003 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:49.507856    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd1aa024-ddd0-46da-8416-b1a69301c1a0-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:51:50.108331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:50.108295    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"
Oct 31 19:51:50.108331 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:50.108321    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp" event={"ID":"dd1aa024-ddd0-46da-8416-b1a69301c1a0","Type":"ContainerDied","Data":"0d77ba5565bb77f101cb0057f3902a396538f0055f8a43a4ab193985933e7cb2"}
Oct 31 19:51:50.108655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:50.108367    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d77ba5565bb77f101cb0057f3902a396538f0055f8a43a4ab193985933e7cb2"
Oct 31 19:51:59.205215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:51:59.205180    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"]
Oct 31 19:52:04.770864 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.770826    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"]
Oct 31 19:52:04.771572 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.771548    5551 cpu_manager.go:414] "containerMap: removing container" podUID="dd1aa024-ddd0-46da-8416-b1a69301c1a0" containerName="revoke-keys"
Oct 31 19:52:04.771696 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.771573    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1aa024-ddd0-46da-8416-b1a69301c1a0" containerName="revoke-keys"
Oct 31 19:52:04.771767 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.771721    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1aa024-ddd0-46da-8416-b1a69301c1a0" containerName="revoke-keys"
Oct 31 19:52:04.781028 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.781004    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:04.849486 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.849268    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"]
Oct 31 19:52:04.857497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.853774    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:04.857497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.853875    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgz5\" (UniqueName: \"kubernetes.io/projected/6b2e51c5-3523-4f90-81fd-bb44989578fb-kube-api-access-fkgz5\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:04.857497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.853940    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:04.876259 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.876230    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:52:04.876488 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.876430    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:52:04.876746 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.876727    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:52:04.882398 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.882375    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:52:04.955137 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.955089    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:04.955377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.955209    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgz5\" (UniqueName: \"kubernetes.io/projected/6b2e51c5-3523-4f90-81fd-bb44989578fb-kube-api-access-fkgz5\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:04.955377 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.955274    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:04.957179 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.957150    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:04.959268 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:04.959243    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:05.014257 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:05.014225    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgz5\" (UniqueName: \"kubernetes.io/projected/6b2e51c5-3523-4f90-81fd-bb44989578fb-kube-api-access-fkgz5\") pod \"maas-api-revoke-sub-61f40ffc-xnrz8\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:05.103405 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:05.103319    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:05.544878 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:05.529261    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"]
Oct 31 19:52:06.234539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:06.234253    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8" event={"ID":"6b2e51c5-3523-4f90-81fd-bb44989578fb","Type":"ContainerStarted","Data":"071c146dd9a106c8f3fa9951720116aeb38bc7a5f42f2c6813457cac6ec80975"}
Oct 31 19:52:06.234539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:06.234322    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8" event={"ID":"6b2e51c5-3523-4f90-81fd-bb44989578fb","Type":"ContainerStarted","Data":"2549752b3cda69e847fa865af18e45b2c4fe2c58e79c101615e5e1fe0ba02532"}
Oct 31 19:52:06.460484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:06.449757    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8" podStartSLOduration=2.44973799 podStartE2EDuration="2.44973799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:06.446840419 +0000 UTC m=+5429.678078721" watchObservedRunningTime="2026-10-31 19:52:06.44973799 +0000 UTC m=+5429.680976281"
Oct 31 19:52:07.245786 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:07.245746    5551 generic.go:356] "Generic (PLEG): container finished" podID="6b2e51c5-3523-4f90-81fd-bb44989578fb" containerID="071c146dd9a106c8f3fa9951720116aeb38bc7a5f42f2c6813457cac6ec80975" exitCode=0
Oct 31 19:52:07.246180 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:07.245953    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8" event={"ID":"6b2e51c5-3523-4f90-81fd-bb44989578fb","Type":"ContainerDied","Data":"071c146dd9a106c8f3fa9951720116aeb38bc7a5f42f2c6813457cac6ec80975"}
Oct 31 19:52:08.400182 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.400158    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:08.502302 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.502214    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2e51c5-3523-4f90-81fd-bb44989578fb-kube-api-access-fkgz5\" (UniqueName: \"kubernetes.io/projected/6b2e51c5-3523-4f90-81fd-bb44989578fb-kube-api-access-fkgz5\") pod \"6b2e51c5-3523-4f90-81fd-bb44989578fb\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") "
Oct 31 19:52:08.502516 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.502322    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-xks-ca\") pod \"6b2e51c5-3523-4f90-81fd-bb44989578fb\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") "
Oct 31 19:52:08.502516 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.502362    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-openshift-ca\") pod \"6b2e51c5-3523-4f90-81fd-bb44989578fb\" (UID: \"6b2e51c5-3523-4f90-81fd-bb44989578fb\") "
Oct 31 19:52:08.502740 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.502716    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-openshift-ca" pod "6b2e51c5-3523-4f90-81fd-bb44989578fb" (UID: "6b2e51c5-3523-4f90-81fd-bb44989578fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:52:08.502812 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.502798    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:08.504545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.504523    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2e51c5-3523-4f90-81fd-bb44989578fb-kube-api-access-fkgz5" pod "6b2e51c5-3523-4f90-81fd-bb44989578fb" (UID: "6b2e51c5-3523-4f90-81fd-bb44989578fb"). InnerVolumeSpecName "kube-api-access-fkgz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:08.504545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.504523    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-xks-ca" pod "6b2e51c5-3523-4f90-81fd-bb44989578fb" (UID: "6b2e51c5-3523-4f90-81fd-bb44989578fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:52:08.604326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.604288    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkgz5\" (UniqueName: \"kubernetes.io/projected/6b2e51c5-3523-4f90-81fd-bb44989578fb-kube-api-access-fkgz5\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:08.604326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:08.604317    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2e51c5-3523-4f90-81fd-bb44989578fb-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:09.259686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:09.259603    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"
Oct 31 19:52:09.259686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:09.259636    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8" event={"ID":"6b2e51c5-3523-4f90-81fd-bb44989578fb","Type":"ContainerDied","Data":"2549752b3cda69e847fa865af18e45b2c4fe2c58e79c101615e5e1fe0ba02532"}
Oct 31 19:52:09.259686 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:09.259682    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2549752b3cda69e847fa865af18e45b2c4fe2c58e79c101615e5e1fe0ba02532"
Oct 31 19:52:14.226497 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.226443    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2" podUID="d76d33d0-994e-431c-895f-600b7da8141e" containerName="main" containerID="cri-o://4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7" gracePeriod=45
Oct 31 19:52:14.481310 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.481257    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:52:14.668930 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.668897    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-tmp-dir\") pod \"d76d33d0-994e-431c-895f-600b7da8141e\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") "
Oct 31 19:52:14.669109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669030    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-home\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-home\") pod \"d76d33d0-994e-431c-895f-600b7da8141e\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") "
Oct 31 19:52:14.669109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669086    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-dshm\") pod \"d76d33d0-994e-431c-895f-600b7da8141e\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") "
Oct 31 19:52:14.669216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669119    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76d33d0-994e-431c-895f-600b7da8141e-kube-api-access-2lv87\" (UniqueName: \"kubernetes.io/projected/d76d33d0-994e-431c-895f-600b7da8141e-kube-api-access-2lv87\") pod \"d76d33d0-994e-431c-895f-600b7da8141e\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") "
Oct 31 19:52:14.669216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669182    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-model-cache\") pod \"d76d33d0-994e-431c-895f-600b7da8141e\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") "
Oct 31 19:52:14.669216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669198    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-tmp-dir" pod "d76d33d0-994e-431c-895f-600b7da8141e" (UID: "d76d33d0-994e-431c-895f-600b7da8141e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:14.669216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669212    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76d33d0-994e-431c-895f-600b7da8141e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d76d33d0-994e-431c-895f-600b7da8141e-tls-certs\") pod \"d76d33d0-994e-431c-895f-600b7da8141e\" (UID: \"d76d33d0-994e-431c-895f-600b7da8141e\") "
Oct 31 19:52:14.669444 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669285    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-home" pod "d76d33d0-994e-431c-895f-600b7da8141e" (UID: "d76d33d0-994e-431c-895f-600b7da8141e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:14.669538 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669506    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-model-cache" pod "d76d33d0-994e-431c-895f-600b7da8141e" (UID: "d76d33d0-994e-431c-895f-600b7da8141e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:14.669678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669660    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:14.669743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669681    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:14.669743 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.669690    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:14.671565 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.671538    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76d33d0-994e-431c-895f-600b7da8141e-tls-certs" pod "d76d33d0-994e-431c-895f-600b7da8141e" (UID: "d76d33d0-994e-431c-895f-600b7da8141e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:52:14.671683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.671608    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-dshm" pod "d76d33d0-994e-431c-895f-600b7da8141e" (UID: "d76d33d0-994e-431c-895f-600b7da8141e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:14.671683 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.671626    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76d33d0-994e-431c-895f-600b7da8141e-kube-api-access-2lv87" pod "d76d33d0-994e-431c-895f-600b7da8141e" (UID: "d76d33d0-994e-431c-895f-600b7da8141e"). InnerVolumeSpecName "kube-api-access-2lv87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:14.770351 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.770294    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76d33d0-994e-431c-895f-600b7da8141e-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:14.770351 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.770318    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76d33d0-994e-431c-895f-600b7da8141e-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:14.770351 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:14.770331    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lv87\" (UniqueName: \"kubernetes.io/projected/d76d33d0-994e-431c-895f-600b7da8141e-kube-api-access-2lv87\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:15.296275 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.296239    5551 generic.go:356] "Generic (PLEG): container finished" podID="d76d33d0-994e-431c-895f-600b7da8141e" containerID="4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7" exitCode=0
Oct 31 19:52:15.296759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.296281    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"
Oct 31 19:52:15.296759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.296292    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2" event={"ID":"d76d33d0-994e-431c-895f-600b7da8141e","Type":"ContainerDied","Data":"4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7"}
Oct 31 19:52:15.296759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.296334    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2" event={"ID":"d76d33d0-994e-431c-895f-600b7da8141e","Type":"ContainerDied","Data":"27b0c3423cedb3a25f4290f0b3e2fc688b2fffeb044433aab62df2cb9452156b"}
Oct 31 19:52:15.296759 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.296355    5551 scope.go:122] "RemoveContainer" containerID="4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7"
Oct 31 19:52:15.308240 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.308220    5551 scope.go:122] "RemoveContainer" containerID="4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7"
Oct 31 19:52:15.308586 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:52:15.308557    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7\": container with ID starting with 4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7 not found: ID does not exist" containerID="4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7"
Oct 31 19:52:15.308676 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.308594    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7"} err="failed to get container status \"4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7\": rpc error: code = NotFound desc = could not find container \"4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7\": container with ID starting with 4329d3050b1e980e7ade417afad58aabe38df0840548923d3986066c46ae2aa7 not found: ID does not exist"
Oct 31 19:52:15.460260 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.460226    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"]
Oct 31 19:52:15.509780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:15.509748    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-869dcdcf5f-kxmr2"]
Oct 31 19:52:16.603659 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:52:16.603627    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:52:16.907886 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:16.907857    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76d33d0-994e-431c-895f-600b7da8141e" path="/var/lib/kubelet/pods/d76d33d0-994e-431c-895f-600b7da8141e/volumes"
Oct 31 19:52:17.844283 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.844024    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"]
Oct 31 19:52:17.845080 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.845055    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6b2e51c5-3523-4f90-81fd-bb44989578fb" containerName="revoke-keys"
Oct 31 19:52:17.845261 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.845246    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2e51c5-3523-4f90-81fd-bb44989578fb" containerName="revoke-keys"
Oct 31 19:52:17.845401 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.845381    5551 cpu_manager.go:414] "containerMap: removing container" podUID="d76d33d0-994e-431c-895f-600b7da8141e" containerName="main"
Oct 31 19:52:17.845524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.845512    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76d33d0-994e-431c-895f-600b7da8141e" containerName="main"
Oct 31 19:52:17.846224 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.846204    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2e51c5-3523-4f90-81fd-bb44989578fb" containerName="revoke-keys"
Oct 31 19:52:17.846364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.846352    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76d33d0-994e-431c-895f-600b7da8141e" containerName="main"
Oct 31 19:52:17.852123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.852099    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898154    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44051314-047d-457f-bbf2-2516d3ae2e38-istiod-ca-cert\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898191    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-istio-token\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898209    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-data\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898349    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-socket\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898399 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898382    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44051314-047d-457f-bbf2-2516d3ae2e38-istio-podinfo\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898400    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-envoy\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898550 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898532    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-credential-socket\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898624 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898570    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-certs\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.898660 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.898636    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvc4\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-kube-api-access-5rvc4\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.904174 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.904138    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-4e15e5-openshift-default-dockercfg-2mbl2\""
Oct 31 19:52:17.956529 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.956498    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"]
Oct 31 19:52:17.999657 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.999616    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-credential-socket\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.999851 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.999676    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-certs\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.999851 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.999729    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvc4\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-kube-api-access-5rvc4\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.999851 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.999802    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44051314-047d-457f-bbf2-2516d3ae2e38-istiod-ca-cert\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:17.999851 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.999826    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-istio-token\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.999875    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-data\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:17.999968    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-socket\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.000049    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-credential-socket\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000090 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.000061    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-certs\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.000099    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44051314-047d-457f-bbf2-2516d3ae2e38-istio-podinfo\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.000121    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-envoy\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000313 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.000252    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-socket\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000537 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.000354    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-data\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.000679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.000658    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44051314-047d-457f-bbf2-2516d3ae2e38-istiod-ca-cert\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.002539 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.002520    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-envoy\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.002728 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.002711    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44051314-047d-457f-bbf2-2516d3ae2e38-istio-podinfo\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.057971 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.057932    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvc4\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-kube-api-access-5rvc4\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.068965 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.068931    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-istio-token\") pod \"e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") " pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:18.174481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:18.173951    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:19.250484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:19.248146    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"]
Oct 31 19:52:19.300383 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:52:19.300316    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44051314_047d_457f_bbf2_2516d3ae2e38.slice/crio-44fe16d41b07397e904d8258c19e4c7f0d167c9c4fff7b085031a264506c0cee WatchSource:0}: Error finding container 44fe16d41b07397e904d8258c19e4c7f0d167c9c4fff7b085031a264506c0cee: Status 404 returned error can't find the container with id 44fe16d41b07397e904d8258c19e4c7f0d167c9c4fff7b085031a264506c0cee
Oct 31 19:52:19.327862 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:19.325715    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:52:19.327862 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:19.325839    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:52:19.327862 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:19.325888    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:52:19.451089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:19.450479    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" event={"ID":"44051314-047d-457f-bbf2-2516d3ae2e38","Type":"ContainerStarted","Data":"44fe16d41b07397e904d8258c19e4c7f0d167c9c4fff7b085031a264506c0cee"}
Oct 31 19:52:19.525898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:19.525554    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:52:19.525898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:19.525620    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:52:20.461679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:20.461643    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" event={"ID":"44051314-047d-457f-bbf2-2516d3ae2e38","Type":"ContainerStarted","Data":"efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5"}
Oct 31 19:52:20.628984 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:20.628916    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" podStartSLOduration=3.628896639 podStartE2EDuration="3.628896639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:20.627031151 +0000 UTC m=+5443.858269442" watchObservedRunningTime="2026-10-31 19:52:20.628896639 +0000 UTC m=+5443.860134929"
Oct 31 19:52:21.174875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:21.174841    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:21.180022 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:21.179995    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:21.468306 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:21.468211    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:21.469132 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:21.469112    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:29.620127 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.620094    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"]
Oct 31 19:52:29.624136 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.624114    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.684802 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.684765    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.684967 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.684904    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.685048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.684971    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62dad-9995-4d65-8e94-86992f23bfda-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.685048 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.684991    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhdh\" (UniqueName: \"kubernetes.io/projected/b1b62dad-9995-4d65-8e94-86992f23bfda-kube-api-access-wmhdh\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.725652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.725620    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:52:29.725652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.725635    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:52:29.725652 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.725640    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4e15e5-serving-cert\""
Oct 31 19:52:29.725979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.725711    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 19:52:29.725979 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.725736    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 19:52:29.729109 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.729083    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"]
Oct 31 19:52:29.786623 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.786590    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62dad-9995-4d65-8e94-86992f23bfda-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.786783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.786631    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhdh\" (UniqueName: \"kubernetes.io/projected/b1b62dad-9995-4d65-8e94-86992f23bfda-kube-api-access-wmhdh\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.786783 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.786682    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.786943 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.786784    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.786943 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:52:29.786880    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-serving-cert: secret "maas-api-e2e-worker-single-main-4e15e5-serving-cert" not found
Oct 31 19:52:29.787015 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:52:29.786979    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls podName:b1b62dad-9995-4d65-8e94-86992f23bfda nodeName:}" failed. No retries permitted until 2026-10-31 19:52:30.286955144 +0000 UTC m=+5453.518193429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls") pod "maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm" (UID: "b1b62dad-9995-4d65-8e94-86992f23bfda") : secret "maas-api-e2e-worker-single-main-4e15e5-serving-cert" not found
Oct 31 19:52:29.787358 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.787337    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62dad-9995-4d65-8e94-86992f23bfda-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.789436 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.789403    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:29.845933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:29.845898    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhdh\" (UniqueName: \"kubernetes.io/projected/b1b62dad-9995-4d65-8e94-86992f23bfda-kube-api-access-wmhdh\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:30.291576 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:30.291535    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:30.294499 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:30.294444    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:30.541095 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:30.541067    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:30.921659 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:30.921632    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"]
Oct 31 19:52:30.924392 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:52:30.924362    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b62dad_9995_4d65_8e94_86992f23bfda.slice/crio-4833d623c07ba466c850e204195c57a1e137949e4e94a30b0808d9ea375fdca8 WatchSource:0}: Error finding container 4833d623c07ba466c850e204195c57a1e137949e4e94a30b0808d9ea375fdca8: Status 404 returned error can't find the container with id 4833d623c07ba466c850e204195c57a1e137949e4e94a30b0808d9ea375fdca8
Oct 31 19:52:30.925762 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:30.925745    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:52:31.528289 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:31.528248    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm" event={"ID":"b1b62dad-9995-4d65-8e94-86992f23bfda","Type":"ContainerStarted","Data":"4833d623c07ba466c850e204195c57a1e137949e4e94a30b0808d9ea375fdca8"}
Oct 31 19:52:32.535688 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:32.535645    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm" event={"ID":"b1b62dad-9995-4d65-8e94-86992f23bfda","Type":"ContainerStarted","Data":"18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f"}
Oct 31 19:52:32.536081 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:32.535779    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:32.650546 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:32.650487    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm" podStartSLOduration=2.754644505 podStartE2EDuration="3.650435251s" totalImagesPullingTime="895.790746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:32.649775265 +0000 UTC m=+5455.881013561" watchObservedRunningTime="2026-10-31 19:52:32.650435251 +0000 UTC m=+5455.881673542"
Oct 31 19:52:38.547328 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:38.547297    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:52:40.926916 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:40.926882    5551 scope.go:122] "RemoveContainer" containerID="ce4ce97410510d5f30d8d5a7e5f30cad7b4ca655a721aad4e1616a5c34dc7fe3"
Oct 31 19:52:40.936255 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:40.936237    5551 scope.go:122] "RemoveContainer" containerID="98fa017c7a6f04b350381aee4946c0e3481a2e1347589c7d46d32e3325c2beef"
Oct 31 19:52:46.612210 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:52:46.612165    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:52:47.659193 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:47.659156    5551 generic.go:356] "Generic (PLEG): container finished" podID="44051314-047d-457f-bbf2-2516d3ae2e38" containerID="efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5" exitCode=0
Oct 31 19:52:47.659667 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:47.659230    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" event={"ID":"44051314-047d-457f-bbf2-2516d3ae2e38","Type":"ContainerDied","Data":"efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5"}
Oct 31 19:52:47.659712 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:47.659675    5551 scope.go:122] "RemoveContainer" containerID="efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5"
Oct 31 19:52:47.660884 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:47.660858    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:52:47.660959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:47.660923    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:52:47.660959 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:47.660955    5551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 19:52:48.680098 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:48.680056    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" event={"ID":"44051314-047d-457f-bbf2-2516d3ae2e38","Type":"ContainerStarted","Data":"639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd"}
Oct 31 19:52:49.174678 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:49.174644    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:49.176266 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:49.176233    5551 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:52:49.176439 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:49.176302    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 19:52:50.178426 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:50.177902    5551 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:52:50.178426 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:50.177965    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 19:52:51.174799 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:51.174740    5551 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:52:51.175001 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:51.174816    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 19:52:51.469719 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:51.469633    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:52.174898 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:52.174856    5551 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:52:52.175130 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:52.174961    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 19:52:53.174953 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:53.174911    5551 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:52:53.175404 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:53.174977    5551 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 19:52:54.179545 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:54.179512    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:52:54.180576 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:52:54.180555    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:53:16.603563 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:53:16.603450    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:53:46.603202 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:53:46.603166    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:53:56.207186 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:56.207145    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"]
Oct 31 19:53:56.207865 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:56.207330    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm" podUID="b1b62dad-9995-4d65-8e94-86992f23bfda" containerName="maas-api" containerID="cri-o://18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f" gracePeriod=30
Oct 31 19:53:57.463030 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.463002    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:53:57.519077 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.519052    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b62dad-9995-4d65-8e94-86992f23bfda-kube-api-access-wmhdh\" (UniqueName: \"kubernetes.io/projected/b1b62dad-9995-4d65-8e94-86992f23bfda-kube-api-access-wmhdh\") pod \"b1b62dad-9995-4d65-8e94-86992f23bfda\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") "
Oct 31 19:53:57.519215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.519106    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-metrics-certs\") pod \"b1b62dad-9995-4d65-8e94-86992f23bfda\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") "
Oct 31 19:53:57.519215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.519150    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls\") pod \"b1b62dad-9995-4d65-8e94-86992f23bfda\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") "
Oct 31 19:53:57.519215 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.519206    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b62dad-9995-4d65-8e94-86992f23bfda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62dad-9995-4d65-8e94-86992f23bfda-openshift-service-ca\") pod \"b1b62dad-9995-4d65-8e94-86992f23bfda\" (UID: \"b1b62dad-9995-4d65-8e94-86992f23bfda\") "
Oct 31 19:53:57.519630 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.519607    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b62dad-9995-4d65-8e94-86992f23bfda-openshift-service-ca" pod "b1b62dad-9995-4d65-8e94-86992f23bfda" (UID: "b1b62dad-9995-4d65-8e94-86992f23bfda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:57.521365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.521340    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-metrics-certs" pod "b1b62dad-9995-4d65-8e94-86992f23bfda" (UID: "b1b62dad-9995-4d65-8e94-86992f23bfda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:53:57.521365 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.521350    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b62dad-9995-4d65-8e94-86992f23bfda-kube-api-access-wmhdh" pod "b1b62dad-9995-4d65-8e94-86992f23bfda" (UID: "b1b62dad-9995-4d65-8e94-86992f23bfda"). InnerVolumeSpecName "kube-api-access-wmhdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:57.521593 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.521381    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls" pod "b1b62dad-9995-4d65-8e94-86992f23bfda" (UID: "b1b62dad-9995-4d65-8e94-86992f23bfda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:53:57.620408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.620328    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhdh\" (UniqueName: \"kubernetes.io/projected/b1b62dad-9995-4d65-8e94-86992f23bfda-kube-api-access-wmhdh\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:53:57.620408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.620357    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:53:57.620408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.620367    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b62dad-9995-4d65-8e94-86992f23bfda-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:53:57.620408 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:57.620376    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62dad-9995-4d65-8e94-86992f23bfda-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:53:58.106513 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.106475    5551 generic.go:356] "Generic (PLEG): container finished" podID="b1b62dad-9995-4d65-8e94-86992f23bfda" containerID="18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f" exitCode=0
Oct 31 19:53:58.106695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.106523    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"
Oct 31 19:53:58.106695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.106568    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm" event={"ID":"b1b62dad-9995-4d65-8e94-86992f23bfda","Type":"ContainerDied","Data":"18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f"}
Oct 31 19:53:58.106695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.106641    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm" event={"ID":"b1b62dad-9995-4d65-8e94-86992f23bfda","Type":"ContainerDied","Data":"4833d623c07ba466c850e204195c57a1e137949e4e94a30b0808d9ea375fdca8"}
Oct 31 19:53:58.106695 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.106672    5551 scope.go:122] "RemoveContainer" containerID="18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f"
Oct 31 19:53:58.117732 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.117715    5551 scope.go:122] "RemoveContainer" containerID="18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f"
Oct 31 19:53:58.118032 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:53:58.118016    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f\": container with ID starting with 18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f not found: ID does not exist" containerID="18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f"
Oct 31 19:53:58.118100 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.118043    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f"} err="failed to get container status \"18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f\": rpc error: code = NotFound desc = could not find container \"18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f\": container with ID starting with 18f98a7e2ba59d772cf95f6e72c2d19f9ad9a519f733b5f3aa14dddcbfd85b0f not found: ID does not exist"
Oct 31 19:53:58.270118 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.270086    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"]
Oct 31 19:53:58.274614 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.274588    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-rrprm"]
Oct 31 19:53:58.909113 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:53:58.909081    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b62dad-9995-4d65-8e94-86992f23bfda" path="/var/lib/kubelet/pods/b1b62dad-9995-4d65-8e94-86992f23bfda/volumes"
Oct 31 19:54:16.604809 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:54:16.604767    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:54:44.834297 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.834210    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"]
Oct 31 19:54:44.834803 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.834784    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b1b62dad-9995-4d65-8e94-86992f23bfda" containerName="maas-api"
Oct 31 19:54:44.834853 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.834806    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b62dad-9995-4d65-8e94-86992f23bfda" containerName="maas-api"
Oct 31 19:54:44.834914 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.834905    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b62dad-9995-4d65-8e94-86992f23bfda" containerName="maas-api"
Oct 31 19:54:44.839609 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.839588    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:44.854124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.854099    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:44.854250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.854139    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lpg\" (UniqueName: \"kubernetes.io/projected/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-kube-api-access-x2lpg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:44.854250 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.854221    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:44.899427 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.899397    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"]
Oct 31 19:54:44.945326 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.945295    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:54:44.945528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.945357    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:54:44.945528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.945410    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:54:44.945528 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.945425    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:54:44.955123 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.955095    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:44.955254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.955143    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lpg\" (UniqueName: \"kubernetes.io/projected/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-kube-api-access-x2lpg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:44.955254 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.955211    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:44.955803 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.955780    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:44.957661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:44.957641    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:45.018560 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:45.018530    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lpg\" (UniqueName: \"kubernetes.io/projected/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-kube-api-access-x2lpg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:45.153049 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:45.153019    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:45.546085 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:45.546052    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"]
Oct 31 19:54:45.547645 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:54:45.547609    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc881d8a_d09f_46f8_bbc7_fcd09c89d9b9.slice/crio-ab426c5254657e34f2a0094a95cf6fe686b9e1d87e54a2cb7b49fc30623f317e WatchSource:0}: Error finding container ab426c5254657e34f2a0094a95cf6fe686b9e1d87e54a2cb7b49fc30623f317e: Status 404 returned error can't find the container with id ab426c5254657e34f2a0094a95cf6fe686b9e1d87e54a2cb7b49fc30623f317e
Oct 31 19:54:46.402065 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:46.402033    5551 generic.go:356] "Generic (PLEG): container finished" podID="dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9" containerID="9b8ef9b0635c5c49538a2de4931f20073bf30c1e0d3ca3b254b71c60018933f5" exitCode=0
Oct 31 19:54:46.402627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:46.402089    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl" event={"ID":"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9","Type":"ContainerDied","Data":"9b8ef9b0635c5c49538a2de4931f20073bf30c1e0d3ca3b254b71c60018933f5"}
Oct 31 19:54:46.402627 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:46.402137    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl" event={"ID":"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9","Type":"ContainerStarted","Data":"ab426c5254657e34f2a0094a95cf6fe686b9e1d87e54a2cb7b49fc30623f317e"}
Oct 31 19:54:46.605070 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:54:46.604992    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:54:47.581531 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.581504    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:47.684933 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.684843    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-xks-ca\") pod \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") "
Oct 31 19:54:47.685097 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.684987    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-openshift-ca\") pod \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") "
Oct 31 19:54:47.685097 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.685017    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-kube-api-access-x2lpg\" (UniqueName: \"kubernetes.io/projected/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-kube-api-access-x2lpg\") pod \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\" (UID: \"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9\") "
Oct 31 19:54:47.685403 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.685364    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-openshift-ca" pod "dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9" (UID: "dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:54:47.687103 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.687080    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-xks-ca" pod "dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9" (UID: "dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:54:47.687253 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.687233    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-kube-api-access-x2lpg" pod "dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9" (UID: "dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9"). InnerVolumeSpecName "kube-api-access-x2lpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:54:47.785858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.785812    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:47.785858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.785853    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2lpg\" (UniqueName: \"kubernetes.io/projected/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-kube-api-access-x2lpg\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:47.785858 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:47.785864    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:48.415524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:48.415487    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"
Oct 31 19:54:48.415524 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:48.415497    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl" event={"ID":"dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9","Type":"ContainerDied","Data":"ab426c5254657e34f2a0094a95cf6fe686b9e1d87e54a2cb7b49fc30623f317e"}
Oct 31 19:54:48.415790 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:54:48.415546    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab426c5254657e34f2a0094a95cf6fe686b9e1d87e54a2cb7b49fc30623f317e"
Oct 31 19:55:04.615238 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:04.615202    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"]
Oct 31 19:55:04.617715 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:04.617690    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e15e5-z4hnl"]
Oct 31 19:55:04.909712 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:04.909680    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9" path="/var/lib/kubelet/pods/dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9/volumes"
Oct 31 19:55:04.958360 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:04.958325    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"]
Oct 31 19:55:04.958605 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:04.958580    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy" containerID="cri-o://639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd" gracePeriod=30
Oct 31 19:55:09.181525 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:09.181484    5551 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:55:09.181977 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:09.181546    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:55:10.214809 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.214777    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:55:10.321034 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.320950    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-data\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321050    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-certs\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321081    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-envoy\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321104    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-istio-token\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-istio-token\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321216 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321133    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44051314-047d-457f-bbf2-2516d3ae2e38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44051314-047d-457f-bbf2-2516d3ae2e38-istiod-ca-cert\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321215    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-kube-api-access-5rvc4\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-kube-api-access-5rvc4\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321271    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-data" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:10.321484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321368    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-socket\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321484 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321416    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-certs" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:10.321726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321422    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-credential-socket\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321552    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44051314-047d-457f-bbf2-2516d3ae2e38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44051314-047d-457f-bbf2-2516d3ae2e38-istio-podinfo\") pod \"44051314-047d-457f-bbf2-2516d3ae2e38\" (UID: \"44051314-047d-457f-bbf2-2516d3ae2e38\") "
Oct 31 19:55:10.321726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321667    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-credential-socket" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:10.321726 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321704    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-socket" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:10.322124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.321907    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44051314-047d-457f-bbf2-2516d3ae2e38-istiod-ca-cert" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:55:10.322124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.322051    5551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.322124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.322067    5551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44051314-047d-457f-bbf2-2516d3ae2e38-istiod-ca-cert\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.322124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.322081    5551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-workload-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.322124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.322096    5551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-credential-socket\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.322124 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.322109    5551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-data\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.323711 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.323674    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-envoy" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:10.323832 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.323766    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-istio-token" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:55:10.324190 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.324167    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-kube-api-access-5rvc4" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "kube-api-access-5rvc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:55:10.324679 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.324648    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44051314-047d-457f-bbf2-2516d3ae2e38-istio-podinfo" pod "44051314-047d-457f-bbf2-2516d3ae2e38" (UID: "44051314-047d-457f-bbf2-2516d3ae2e38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:55:10.422952 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.422922    5551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44051314-047d-457f-bbf2-2516d3ae2e38-istio-envoy\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.422952 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.422950    5551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-istio-token\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.423158 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.422960    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rvc4\" (UniqueName: \"kubernetes.io/projected/44051314-047d-457f-bbf2-2516d3ae2e38-kube-api-access-5rvc4\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.423158 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.422972    5551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44051314-047d-457f-bbf2-2516d3ae2e38-istio-podinfo\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:10.546219 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.546187    5551 generic.go:356] "Generic (PLEG): container finished" podID="44051314-047d-457f-bbf2-2516d3ae2e38" containerID="639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd" exitCode=0
Oct 31 19:55:10.546393 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.546243    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"
Oct 31 19:55:10.546393 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.546270    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" event={"ID":"44051314-047d-457f-bbf2-2516d3ae2e38","Type":"ContainerDied","Data":"639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd"}
Oct 31 19:55:10.546393 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.546324    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r" event={"ID":"44051314-047d-457f-bbf2-2516d3ae2e38","Type":"ContainerDied","Data":"44fe16d41b07397e904d8258c19e4c7f0d167c9c4fff7b085031a264506c0cee"}
Oct 31 19:55:10.546393 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.546346    5551 scope.go:122] "RemoveContainer" containerID="639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd"
Oct 31 19:55:10.556852 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.556835    5551 scope.go:122] "RemoveContainer" containerID="efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5"
Oct 31 19:55:10.565415 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.565397    5551 scope.go:122] "RemoveContainer" containerID="639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd"
Oct 31 19:55:10.565715 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:55:10.565697    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd\": container with ID starting with 639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd not found: ID does not exist" containerID="639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd"
Oct 31 19:55:10.565797 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.565723    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd"} err="failed to get container status \"639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd\": rpc error: code = NotFound desc = could not find container \"639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd\": container with ID starting with 639fffce0380e547b2933ce9041626f0df63b9ffb9b6a2592bf8c9a1a042fcdd not found: ID does not exist"
Oct 31 19:55:10.565797 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.565738    5551 scope.go:122] "RemoveContainer" containerID="efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5"
Oct 31 19:55:10.565962 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:55:10.565947    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5\": container with ID starting with efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5 not found: ID does not exist" containerID="efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5"
Oct 31 19:55:10.566011 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.565967    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5"} err="failed to get container status \"efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5\": rpc error: code = NotFound desc = could not find container \"efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5\": container with ID starting with efea81f7faaf0e87b966827effb28825a3e19672ae34b751c54a7dc6dc9162e5 not found: ID does not exist"
Oct 31 19:55:10.715010 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.714978    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"]
Oct 31 19:55:10.717051 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.717031    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e15e5-openshift-default-668d749bbf5rk2r"]
Oct 31 19:55:10.908532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:10.908503    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" path="/var/lib/kubelet/pods/44051314-047d-457f-bbf2-2516d3ae2e38/volumes"
Oct 31 19:55:11.921113 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.921074    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"]
Oct 31 19:55:11.921691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.921519    5551 cpu_manager.go:414] "containerMap: removing container" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy"
Oct 31 19:55:11.921691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.921532    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy"
Oct 31 19:55:11.921691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.921566    5551 cpu_manager.go:414] "containerMap: removing container" podUID="dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9" containerName="revoke-keys"
Oct 31 19:55:11.921691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.921572    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9" containerName="revoke-keys"
Oct 31 19:55:11.921691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.921640    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc881d8a-d09f-46f8-bbc7-fcd09c89d9b9" containerName="revoke-keys"
Oct 31 19:55:11.921691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.921652    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy"
Oct 31 19:55:11.921691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.921659    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy"
Oct 31 19:55:11.926134 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:11.926113    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.029005 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.028972    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-20b73ef4-kserve-self-signed-certs\""
Oct 31 19:55:12.031307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.031287    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"]
Oct 31 19:55:12.038636 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.038613    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-model-cache\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.038710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.038661    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-home\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.038710 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.038679    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmb9\" (UniqueName: \"kubernetes.io/projected/23031ea8-c36e-4021-8a87-01c4983afa2f-kube-api-access-kfmb9\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.038791 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.038781    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-dshm\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.038829 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.038807    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-tmp-dir\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.038868 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.038844    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23031ea8-c36e-4021-8a87-01c4983afa2f-tls-certs\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140120 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140082    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-model-cache\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140134    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-home\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140153    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmb9\" (UniqueName: \"kubernetes.io/projected/23031ea8-c36e-4021-8a87-01c4983afa2f-kube-api-access-kfmb9\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140170    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-dshm\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140189    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-tmp-dir\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140320 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140218    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23031ea8-c36e-4021-8a87-01c4983afa2f-tls-certs\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140661 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140635    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-model-cache\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140725 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140693    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-tmp-dir\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.140784 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.140729    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-home\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.142697 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.142674    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-dshm\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.143062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.143040    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23031ea8-c36e-4021-8a87-01c4983afa2f-tls-certs\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.197879 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.197795    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmb9\" (UniqueName: \"kubernetes.io/projected/23031ea8-c36e-4021-8a87-01c4983afa2f-kube-api-access-kfmb9\") pod \"e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") " pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.238440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.238402    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:12.626440 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:12.626417    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"]
Oct 31 19:55:12.628539 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:55:12.628511    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23031ea8_c36e_4021_8a87_01c4983afa2f.slice/crio-bdec891f7441880baafc51b3f05fa11c63acd47001274764e66dc8bfeb9424e5 WatchSource:0}: Error finding container bdec891f7441880baafc51b3f05fa11c63acd47001274764e66dc8bfeb9424e5: Status 404 returned error can't find the container with id bdec891f7441880baafc51b3f05fa11c63acd47001274764e66dc8bfeb9424e5
Oct 31 19:55:13.571525 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:13.571484    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v" event={"ID":"23031ea8-c36e-4021-8a87-01c4983afa2f","Type":"ContainerStarted","Data":"58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986"}
Oct 31 19:55:13.571929 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:13.571535    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v" event={"ID":"23031ea8-c36e-4021-8a87-01c4983afa2f","Type":"ContainerStarted","Data":"bdec891f7441880baafc51b3f05fa11c63acd47001274764e66dc8bfeb9424e5"}
Oct 31 19:55:13.686926 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:13.686855    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v" podStartSLOduration=2.686836435 podStartE2EDuration="2.686836435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:13.684940802 +0000 UTC m=+5616.916179093" watchObservedRunningTime="2026-10-31 19:55:13.686836435 +0000 UTC m=+5616.918074726"
Oct 31 19:55:14.239262 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:14.239223    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:16.604755 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:55:16.604717    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:55:22.238817 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:22.238736    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:22.253128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:22.253099    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:22.687328 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:22.687300    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:55:46.620122 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:55:46.620077    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:55:47.720669 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:47.720621    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:55:47.721342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:47.721285    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:47.776613 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:47.776572    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:55:47.776935 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:55:47.776904    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:56:16.604440 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:56:16.604349    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:56:21.471807 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:21.471764    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"]
Oct 31 19:56:36.492015 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.491908    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v" podUID="23031ea8-c36e-4021-8a87-01c4983afa2f" containerName="main" containerID="cri-o://58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986" gracePeriod=45
Oct 31 19:56:36.743478 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.743391    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:56:36.803160 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803130    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-home\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-home\") pod \"23031ea8-c36e-4021-8a87-01c4983afa2f\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") "
Oct 31 19:56:36.803370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803247    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23031ea8-c36e-4021-8a87-01c4983afa2f-kube-api-access-kfmb9\" (UniqueName: \"kubernetes.io/projected/23031ea8-c36e-4021-8a87-01c4983afa2f-kube-api-access-kfmb9\") pod \"23031ea8-c36e-4021-8a87-01c4983afa2f\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") "
Oct 31 19:56:36.803370 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803319    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-model-cache\") pod \"23031ea8-c36e-4021-8a87-01c4983afa2f\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") "
Oct 31 19:56:36.803532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803367    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-dshm\") pod \"23031ea8-c36e-4021-8a87-01c4983afa2f\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") "
Oct 31 19:56:36.803532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803431    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-tmp-dir\") pod \"23031ea8-c36e-4021-8a87-01c4983afa2f\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") "
Oct 31 19:56:36.803532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803444    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-home" pod "23031ea8-c36e-4021-8a87-01c4983afa2f" (UID: "23031ea8-c36e-4021-8a87-01c4983afa2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:36.803532 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803497    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23031ea8-c36e-4021-8a87-01c4983afa2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/23031ea8-c36e-4021-8a87-01c4983afa2f-tls-certs\") pod \"23031ea8-c36e-4021-8a87-01c4983afa2f\" (UID: \"23031ea8-c36e-4021-8a87-01c4983afa2f\") "
Oct 31 19:56:36.803737 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803590    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-model-cache" pod "23031ea8-c36e-4021-8a87-01c4983afa2f" (UID: "23031ea8-c36e-4021-8a87-01c4983afa2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:36.803804 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803772    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-tmp-dir" pod "23031ea8-c36e-4021-8a87-01c4983afa2f" (UID: "23031ea8-c36e-4021-8a87-01c4983afa2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:36.803991 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.803967    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:36.804292 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.804012    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:36.804292 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.804027    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:36.805715 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.805688    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-dshm" pod "23031ea8-c36e-4021-8a87-01c4983afa2f" (UID: "23031ea8-c36e-4021-8a87-01c4983afa2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:36.805822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.805737    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23031ea8-c36e-4021-8a87-01c4983afa2f-tls-certs" pod "23031ea8-c36e-4021-8a87-01c4983afa2f" (UID: "23031ea8-c36e-4021-8a87-01c4983afa2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:36.805822 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.805748    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23031ea8-c36e-4021-8a87-01c4983afa2f-kube-api-access-kfmb9" pod "23031ea8-c36e-4021-8a87-01c4983afa2f" (UID: "23031ea8-c36e-4021-8a87-01c4983afa2f"). InnerVolumeSpecName "kube-api-access-kfmb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:36.904670 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.904634    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfmb9\" (UniqueName: \"kubernetes.io/projected/23031ea8-c36e-4021-8a87-01c4983afa2f-kube-api-access-kfmb9\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:36.904670 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.904667    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23031ea8-c36e-4021-8a87-01c4983afa2f-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:36.904670 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:36.904677    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23031ea8-c36e-4021-8a87-01c4983afa2f-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:37.216613 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.216577    5551 generic.go:356] "Generic (PLEG): container finished" podID="23031ea8-c36e-4021-8a87-01c4983afa2f" containerID="58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986" exitCode=0
Oct 31 19:56:37.216798 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.216628    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"
Oct 31 19:56:37.216798 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.216660    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v" event={"ID":"23031ea8-c36e-4021-8a87-01c4983afa2f","Type":"ContainerDied","Data":"58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986"}
Oct 31 19:56:37.216798 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.216702    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v" event={"ID":"23031ea8-c36e-4021-8a87-01c4983afa2f","Type":"ContainerDied","Data":"bdec891f7441880baafc51b3f05fa11c63acd47001274764e66dc8bfeb9424e5"}
Oct 31 19:56:37.216798 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.216720    5551 scope.go:122] "RemoveContainer" containerID="58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986"
Oct 31 19:56:37.225888 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.225871    5551 scope.go:122] "RemoveContainer" containerID="58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986"
Oct 31 19:56:37.226158 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:56:37.226136    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986\": container with ID starting with 58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986 not found: ID does not exist" containerID="58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986"
Oct 31 19:56:37.226232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.226171    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986"} err="failed to get container status \"58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986\": rpc error: code = NotFound desc = could not find container \"58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986\": container with ID starting with 58660af040689688f70c8000fdd59fb48c772046ecc04f32f58e7ef242013986 not found: ID does not exist"
Oct 31 19:56:37.379707 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.379669    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"]
Oct 31 19:56:37.381851 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.381826    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v"]
Oct 31 19:56:37.677752 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.677712    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-20b73ef4-kserve-759bbf7789-rx98v" podUID="23031ea8-c36e-4021-8a87-01c4983afa2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.247:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:56:37.940369 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.940288    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"]
Oct 31 19:56:37.940824 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.940806    5551 cpu_manager.go:414] "containerMap: removing container" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy"
Oct 31 19:56:37.940887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.940827    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44051314-047d-457f-bbf2-2516d3ae2e38" containerName="istio-proxy"
Oct 31 19:56:37.940887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.940855    5551 cpu_manager.go:414] "containerMap: removing container" podUID="23031ea8-c36e-4021-8a87-01c4983afa2f" containerName="main"
Oct 31 19:56:37.940887 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.940860    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23031ea8-c36e-4021-8a87-01c4983afa2f" containerName="main"
Oct 31 19:56:37.941006 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.940925    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="23031ea8-c36e-4021-8a87-01c4983afa2f" containerName="main"
Oct 31 19:56:37.945513 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.945492    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:37.998893 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.998824    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:56:37.999309 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.999290    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:56:37.999425 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:37.999337    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:56:38.001834 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.001810    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"]
Oct 31 19:56:38.016951 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.016927    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.017069 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.016962    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.017188 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.017163    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxfw\" (UniqueName: \"kubernetes.io/projected/b35d2e76-956a-497d-b322-132f7c3a9bf8-kube-api-access-tmxfw\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.047364 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.047336    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:56:38.118540 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.118513    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.118655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.118548    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.118655 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.118633    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxfw\" (UniqueName: \"kubernetes.io/projected/b35d2e76-956a-497d-b322-132f7c3a9bf8-kube-api-access-tmxfw\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.119178 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.119150    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.120992 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.120970    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.175962 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.175936    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxfw\" (UniqueName: \"kubernetes.io/projected/b35d2e76-956a-497d-b322-132f7c3a9bf8-kube-api-access-tmxfw\") pod \"maas-api-revoke-sub-3e31a506-s6hnd\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.258874 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.258813    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:38.637691 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.637664    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"]
Oct 31 19:56:38.639679 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:56:38.639650    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35d2e76_956a_497d_b322_132f7c3a9bf8.slice/crio-7bb0c0e10d8234e498deb4e162d1e96e04e1dac44a99dc83876eba58b7a3b8d1 WatchSource:0}: Error finding container 7bb0c0e10d8234e498deb4e162d1e96e04e1dac44a99dc83876eba58b7a3b8d1: Status 404 returned error can't find the container with id 7bb0c0e10d8234e498deb4e162d1e96e04e1dac44a99dc83876eba58b7a3b8d1
Oct 31 19:56:38.909217 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:38.909139    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23031ea8-c36e-4021-8a87-01c4983afa2f" path="/var/lib/kubelet/pods/23031ea8-c36e-4021-8a87-01c4983afa2f/volumes"
Oct 31 19:56:39.231625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:39.231526    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd" event={"ID":"b35d2e76-956a-497d-b322-132f7c3a9bf8","Type":"ContainerStarted","Data":"53abd3b24006f6d2aab0d2377f51df185d153d8d84d3339e9c323088d7513a18"}
Oct 31 19:56:39.231625 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:39.231597    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd" event={"ID":"b35d2e76-956a-497d-b322-132f7c3a9bf8","Type":"ContainerStarted","Data":"7bb0c0e10d8234e498deb4e162d1e96e04e1dac44a99dc83876eba58b7a3b8d1"}
Oct 31 19:56:39.366089 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:39.366031    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd" podStartSLOduration=2.366013282 podStartE2EDuration="2.366013282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:39.355911962 +0000 UTC m=+5702.587150256" watchObservedRunningTime="2026-10-31 19:56:39.366013282 +0000 UTC m=+5702.597251593"
Oct 31 19:56:40.247193 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:40.247157    5551 generic.go:356] "Generic (PLEG): container finished" podID="b35d2e76-956a-497d-b322-132f7c3a9bf8" containerID="53abd3b24006f6d2aab0d2377f51df185d153d8d84d3339e9c323088d7513a18" exitCode=0
Oct 31 19:56:40.247882 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:40.247292    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd" event={"ID":"b35d2e76-956a-497d-b322-132f7c3a9bf8","Type":"ContainerDied","Data":"53abd3b24006f6d2aab0d2377f51df185d153d8d84d3339e9c323088d7513a18"}
Oct 31 19:56:40.977426 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:40.977398    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:56:40.984673 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:40.984587    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 19:56:42.128481 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.127033    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:42.301780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.301741    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd" event={"ID":"b35d2e76-956a-497d-b322-132f7c3a9bf8","Type":"ContainerDied","Data":"7bb0c0e10d8234e498deb4e162d1e96e04e1dac44a99dc83876eba58b7a3b8d1"}
Oct 31 19:56:42.302171 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.302152    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bb0c0e10d8234e498deb4e162d1e96e04e1dac44a99dc83876eba58b7a3b8d1"
Oct 31 19:56:42.302307 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.302100    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"
Oct 31 19:56:42.315072 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.315027    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-openshift-ca\") pod \"b35d2e76-956a-497d-b322-132f7c3a9bf8\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") "
Oct 31 19:56:42.315232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.315133    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-xks-ca\") pod \"b35d2e76-956a-497d-b322-132f7c3a9bf8\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") "
Oct 31 19:56:42.315232 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.315170    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35d2e76-956a-497d-b322-132f7c3a9bf8-kube-api-access-tmxfw\" (UniqueName: \"kubernetes.io/projected/b35d2e76-956a-497d-b322-132f7c3a9bf8-kube-api-access-tmxfw\") pod \"b35d2e76-956a-497d-b322-132f7c3a9bf8\" (UID: \"b35d2e76-956a-497d-b322-132f7c3a9bf8\") "
Oct 31 19:56:42.315542 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.315513    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-openshift-ca" pod "b35d2e76-956a-497d-b322-132f7c3a9bf8" (UID: "b35d2e76-956a-497d-b322-132f7c3a9bf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:42.315645 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.315628    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:42.317662 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.317632    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-xks-ca" pod "b35d2e76-956a-497d-b322-132f7c3a9bf8" (UID: "b35d2e76-956a-497d-b322-132f7c3a9bf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:42.318575 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.318547    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35d2e76-956a-497d-b322-132f7c3a9bf8-kube-api-access-tmxfw" pod "b35d2e76-956a-497d-b322-132f7c3a9bf8" (UID: "b35d2e76-956a-497d-b322-132f7c3a9bf8"). InnerVolumeSpecName "kube-api-access-tmxfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:42.416432 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.416394    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35d2e76-956a-497d-b322-132f7c3a9bf8-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:42.416432 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:42.416427    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxfw\" (UniqueName: \"kubernetes.io/projected/b35d2e76-956a-497d-b322-132f7c3a9bf8-kube-api-access-tmxfw\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:46.603662 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:56:46.603622    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:56:50.046093 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:50.046053    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"]
Oct 31 19:56:50.095191 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:50.095160    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6c26f0e-cjgwp"]
Oct 31 19:56:50.908415 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:56:50.908381    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1aa024-ddd0-46da-8416-b1a69301c1a0" path="/var/lib/kubelet/pods/dd1aa024-ddd0-46da-8416-b1a69301c1a0/volumes"
Oct 31 19:57:09.052976 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:57:09.052943    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"]
Oct 31 19:57:09.102311 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:57:09.102274    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f40ffc-xnrz8"]
Oct 31 19:57:10.908615 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:57:10.908578    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2e51c5-3523-4f90-81fd-bb44989578fb" path="/var/lib/kubelet/pods/6b2e51c5-3523-4f90-81fd-bb44989578fb/volumes"
Oct 31 19:57:16.604239 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:57:16.604201    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:57:46.603016 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:57:46.602972    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:58:16.604143 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:58:16.604105    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:58:41.195564 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:58:41.195530    5551 scope.go:122] "RemoveContainer" containerID="071c146dd9a106c8f3fa9951720116aeb38bc7a5f42f2c6813457cac6ec80975"
Oct 31 19:58:41.205111 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:58:41.205088    5551 scope.go:122] "RemoveContainer" containerID="39748b6af24ef48e7b837a77533ae4acbb13bd1e423d03cdb1e3d5309e8f4902"
Oct 31 19:58:46.603174 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:58:46.603140    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:59:16.603816 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:59:16.603742    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:59:45.421505 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.421471    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"]
Oct 31 19:59:45.422122 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.422100    5551 cpu_manager.go:414] "containerMap: removing container" podUID="b35d2e76-956a-497d-b322-132f7c3a9bf8" containerName="revoke-keys"
Oct 31 19:59:45.422218 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.422125    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35d2e76-956a-497d-b322-132f7c3a9bf8" containerName="revoke-keys"
Oct 31 19:59:45.422291 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.422277    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35d2e76-956a-497d-b322-132f7c3a9bf8" containerName="revoke-keys"
Oct 31 19:59:45.425915 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.425893    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.481878 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.481855    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"]
Oct 31 19:59:45.528235 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.528207    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:59:45.528363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.528259    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:45.528363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.528297    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:59:45.528363 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.528315    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:59:45.604205 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.604176    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.604336 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.604209    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4827l\" (UniqueName: \"kubernetes.io/projected/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-kube-api-access-4827l\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.604336 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.604301    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.705970 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.705862    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.706235 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.706021    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.706235 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.706055    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4827l\" (UniqueName: \"kubernetes.io/projected/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-kube-api-access-4827l\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.706689 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.706666    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.708582 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.708557    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:45.763780 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:45.763741    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4827l\" (UniqueName: \"kubernetes.io/projected/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-kube-api-access-4827l\") pod \"maas-api-revoke-sub-e2547622-4pndc\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:46.038724 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:46.038642    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:46.419520 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:46.419493    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"]
Oct 31 19:59:46.421593 ip-10-0-132-99 kubenswrapper[5551]: W1031 19:59:46.421565    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa10918_6cf6_4d2d_8b7c_2a9717c86d72.slice/crio-71c6222a0d3bb7407252b385e27a89cba9d1021f250a0ed4c0eb4508a6140231 WatchSource:0}: Error finding container 71c6222a0d3bb7407252b385e27a89cba9d1021f250a0ed4c0eb4508a6140231: Status 404 returned error can't find the container with id 71c6222a0d3bb7407252b385e27a89cba9d1021f250a0ed4c0eb4508a6140231
Oct 31 19:59:46.431926 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:46.431900    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc" event={"ID":"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72","Type":"ContainerStarted","Data":"71c6222a0d3bb7407252b385e27a89cba9d1021f250a0ed4c0eb4508a6140231"}
Oct 31 19:59:46.603007 ip-10-0-132-99 kubenswrapper[5551]: E1031 19:59:46.602972    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 19:59:47.438875 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:47.438845    5551 generic.go:356] "Generic (PLEG): container finished" podID="1aa10918-6cf6-4d2d-8b7c-2a9717c86d72" containerID="af8f628232a3f195a300ee47dedd062608bd0a494e9d7f0cb5d6741cd51a8bf3" exitCode=0
Oct 31 19:59:47.439281 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:47.438930    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc" event={"ID":"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72","Type":"ContainerDied","Data":"af8f628232a3f195a300ee47dedd062608bd0a494e9d7f0cb5d6741cd51a8bf3"}
Oct 31 19:59:48.589062 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.589037    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:48.735657 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.735574    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-openshift-ca\") pod \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") "
Oct 31 19:59:48.735825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.735721    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-kube-api-access-4827l\" (UniqueName: \"kubernetes.io/projected/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-kube-api-access-4827l\") pod \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") "
Oct 31 19:59:48.735825 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.735754    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-xks-ca\") pod \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\" (UID: \"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72\") "
Oct 31 19:59:48.735953 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.735934    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-openshift-ca" pod "1aa10918-6cf6-4d2d-8b7c-2a9717c86d72" (UID: "1aa10918-6cf6-4d2d-8b7c-2a9717c86d72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:48.736157 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.736141    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:59:48.737910 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.737882    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-xks-ca" pod "1aa10918-6cf6-4d2d-8b7c-2a9717c86d72" (UID: "1aa10918-6cf6-4d2d-8b7c-2a9717c86d72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:48.738029 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.738015    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-kube-api-access-4827l" pod "1aa10918-6cf6-4d2d-8b7c-2a9717c86d72" (UID: "1aa10918-6cf6-4d2d-8b7c-2a9717c86d72"). InnerVolumeSpecName "kube-api-access-4827l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:48.836974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.836944    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4827l\" (UniqueName: \"kubernetes.io/projected/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-kube-api-access-4827l\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:59:48.836974 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:48.836970    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:59:49.451128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:49.451090    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"
Oct 31 19:59:49.451128 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:49.451111    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc" event={"ID":"1aa10918-6cf6-4d2d-8b7c-2a9717c86d72","Type":"ContainerDied","Data":"71c6222a0d3bb7407252b385e27a89cba9d1021f250a0ed4c0eb4508a6140231"}
Oct 31 19:59:49.451342 ip-10-0-132-99 kubenswrapper[5551]: I1031 19:59:49.451156    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71c6222a0d3bb7407252b385e27a89cba9d1021f250a0ed4c0eb4508a6140231"
Oct 31 20:00:16.334841 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.334808    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"]
Oct 31 20:00:16.335269 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.335256    5551 cpu_manager.go:414] "containerMap: removing container" podUID="1aa10918-6cf6-4d2d-8b7c-2a9717c86d72" containerName="revoke-keys"
Oct 31 20:00:16.335320 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.335270    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa10918-6cf6-4d2d-8b7c-2a9717c86d72" containerName="revoke-keys"
Oct 31 20:00:16.335374 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.335356    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa10918-6cf6-4d2d-8b7c-2a9717c86d72" containerName="revoke-keys"
Oct 31 20:00:16.338684 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.338665    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.392627 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.392582    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:00:16.392627 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.392596    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:00:16.397814 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.397785    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.398093 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.398056    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhh88\" (UniqueName: \"kubernetes.io/projected/081b1715-0413-40c2-bf21-455ec593fec9-kube-api-access-qhh88\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.398206 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.398114    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.441265 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.441237    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:00:16.441265 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.441247    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:00:16.443725 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.443704    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"]
Oct 31 20:00:16.499086 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.499046    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.499296 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.499131    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhh88\" (UniqueName: \"kubernetes.io/projected/081b1715-0413-40c2-bf21-455ec593fec9-kube-api-access-qhh88\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.499296 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.499161    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.499857 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.499825    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.502187 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.502161    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.556353 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.556321    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhh88\" (UniqueName: \"kubernetes.io/projected/081b1715-0413-40c2-bf21-455ec593fec9-kube-api-access-qhh88\") pod \"maas-api-revoke-sub-c1e271a9-lgxd4\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:16.605193 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:00:16.605101    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:00:16.651425 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:16.651397    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:17.028280 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:17.028253    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"]
Oct 31 20:00:17.030673 ip-10-0-132-99 kubenswrapper[5551]: W1031 20:00:17.030646    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081b1715_0413_40c2_bf21_455ec593fec9.slice/crio-8abbb86e2dda736d68dfb6d627f411c1fa13eaaa2c324f9ca7993a9527e7b895 WatchSource:0}: Error finding container 8abbb86e2dda736d68dfb6d627f411c1fa13eaaa2c324f9ca7993a9527e7b895: Status 404 returned error can't find the container with id 8abbb86e2dda736d68dfb6d627f411c1fa13eaaa2c324f9ca7993a9527e7b895
Oct 31 20:00:17.746242 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:17.746203    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4" event={"ID":"081b1715-0413-40c2-bf21-455ec593fec9","Type":"ContainerStarted","Data":"edcd5a272307c9920c3abe764551288937c62268a48a79f3a99266e07ad34e65"}
Oct 31 20:00:17.746705 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:17.746267    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4" event={"ID":"081b1715-0413-40c2-bf21-455ec593fec9","Type":"ContainerStarted","Data":"8abbb86e2dda736d68dfb6d627f411c1fa13eaaa2c324f9ca7993a9527e7b895"}
Oct 31 20:00:17.862581 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:17.862530    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4" podStartSLOduration=1.862514722 podStartE2EDuration="1.862514722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:17.861202552 +0000 UTC m=+5921.092440853" watchObservedRunningTime="2026-10-31 20:00:17.862514722 +0000 UTC m=+5921.093753011"
Oct 31 20:00:18.753128 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:18.753097    5551 generic.go:356] "Generic (PLEG): container finished" podID="081b1715-0413-40c2-bf21-455ec593fec9" containerID="edcd5a272307c9920c3abe764551288937c62268a48a79f3a99266e07ad34e65" exitCode=0
Oct 31 20:00:18.753638 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:18.753188    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4" event={"ID":"081b1715-0413-40c2-bf21-455ec593fec9","Type":"ContainerDied","Data":"edcd5a272307c9920c3abe764551288937c62268a48a79f3a99266e07ad34e65"}
Oct 31 20:00:19.901964 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:19.901943    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:19.930758 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:19.930732    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-openshift-ca\") pod \"081b1715-0413-40c2-bf21-455ec593fec9\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") "
Oct 31 20:00:19.930877 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:19.930822    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-xks-ca\") pod \"081b1715-0413-40c2-bf21-455ec593fec9\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") "
Oct 31 20:00:19.930877 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:19.930847    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081b1715-0413-40c2-bf21-455ec593fec9-kube-api-access-qhh88\" (UniqueName: \"kubernetes.io/projected/081b1715-0413-40c2-bf21-455ec593fec9-kube-api-access-qhh88\") pod \"081b1715-0413-40c2-bf21-455ec593fec9\" (UID: \"081b1715-0413-40c2-bf21-455ec593fec9\") "
Oct 31 20:00:19.931096 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:19.931070    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-openshift-ca" pod "081b1715-0413-40c2-bf21-455ec593fec9" (UID: "081b1715-0413-40c2-bf21-455ec593fec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:19.932955 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:19.932932    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-xks-ca" pod "081b1715-0413-40c2-bf21-455ec593fec9" (UID: "081b1715-0413-40c2-bf21-455ec593fec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:19.933090 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:19.933073    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081b1715-0413-40c2-bf21-455ec593fec9-kube-api-access-qhh88" pod "081b1715-0413-40c2-bf21-455ec593fec9" (UID: "081b1715-0413-40c2-bf21-455ec593fec9"). InnerVolumeSpecName "kube-api-access-qhh88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:20.032113 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:20.032026    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.032113 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:20.032067    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081b1715-0413-40c2-bf21-455ec593fec9-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.032113 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:20.032083    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhh88\" (UniqueName: \"kubernetes.io/projected/081b1715-0413-40c2-bf21-455ec593fec9-kube-api-access-qhh88\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.768131 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:20.768095    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"
Oct 31 20:00:20.768131 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:20.768126    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4" event={"ID":"081b1715-0413-40c2-bf21-455ec593fec9","Type":"ContainerDied","Data":"8abbb86e2dda736d68dfb6d627f411c1fa13eaaa2c324f9ca7993a9527e7b895"}
Oct 31 20:00:20.768474 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:20.768208    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8abbb86e2dda736d68dfb6d627f411c1fa13eaaa2c324f9ca7993a9527e7b895"
Oct 31 20:00:36.422485 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:36.421588    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:00:36.422485 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:00:36.421677    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:00:46.609931 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:00:46.609844    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:01:16.604337 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:01:16.604290    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:01:41.248868 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:01:41.248831    5551 scope.go:122] "RemoveContainer" containerID="9b8ef9b0635c5c49538a2de4931f20073bf30c1e0d3ca3b254b71c60018933f5"
Oct 31 20:01:41.269937 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:01:41.269916    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 20:01:41.323915 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:01:41.323886    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 20:01:42.045676 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:01:42.045645    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"]
Oct 31 20:01:42.096051 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:01:42.096012    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s6hnd"]
Oct 31 20:01:42.914846 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:01:42.914798    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35d2e76-956a-497d-b322-132f7c3a9bf8" path="/var/lib/kubelet/pods/b35d2e76-956a-497d-b322-132f7c3a9bf8/volumes"
Oct 31 20:01:46.606145 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:01:46.606111    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:02:02.714807 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:02:02.714630    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:02:02.714807 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:02:02.714713    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:02:16.603738 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:02:16.603656    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:02:41.274870 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:02:41.274839    5551 scope.go:122] "RemoveContainer" containerID="53abd3b24006f6d2aab0d2377f51df185d153d8d84d3339e9c323088d7513a18"
Oct 31 20:02:46.605096 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:02:46.605043    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:03:16.605528 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:03:16.605395    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:03:46.821295 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:03:46.821043    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:03:47.723626 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:03:47.723584    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:03:47.724001 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:03:47.723965    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:03:47.777613 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:03:47.777573    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:03:47.777907 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:03:47.777869    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:04:16.604025 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:04:16.603987    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:04:46.606330 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:04:46.606293    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:04:49.074484 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:04:49.064614    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"]
Oct 31 20:04:49.111040 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:04:49.110972    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-4pndc"]
Oct 31 20:04:49.529638 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:04:49.529594    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:04:49.530044 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:04:49.529999    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:04:50.907871 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:04:50.907840    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa10918-6cf6-4d2d-8b7c-2a9717c86d72" path="/var/lib/kubelet/pods/1aa10918-6cf6-4d2d-8b7c-2a9717c86d72/volumes"
Oct 31 20:05:16.604375 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:05:16.604283    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:05:20.045305 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:20.045271    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"]
Oct 31 20:05:20.094246 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:20.094220    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lgxd4"]
Oct 31 20:05:20.908349 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:20.908317    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081b1715-0413-40c2-bf21-455ec593fec9" path="/var/lib/kubelet/pods/081b1715-0413-40c2-bf21-455ec593fec9/volumes"
Oct 31 20:05:28.536596 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.536556    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"]
Oct 31 20:05:28.537098 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.537066    5551 cpu_manager.go:414] "containerMap: removing container" podUID="081b1715-0413-40c2-bf21-455ec593fec9" containerName="revoke-keys"
Oct 31 20:05:28.537098 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.537079    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081b1715-0413-40c2-bf21-455ec593fec9" containerName="revoke-keys"
Oct 31 20:05:28.537184 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.537170    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="081b1715-0413-40c2-bf21-455ec593fec9" containerName="revoke-keys"
Oct 31 20:05:28.540596 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.540563    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.592940 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.592899    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-42030c-serving-cert\""
Oct 31 20:05:28.593121 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.593064    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 20:05:28.642632 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.642596    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:05:28.642825 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.642654    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:05:28.645352 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.645327    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"]
Oct 31 20:05:28.652129 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.652109    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.652196 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.652153    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.652261 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.652236    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc6tq\" (UniqueName: \"kubernetes.io/projected/571064f9-1393-43e1-a943-dd1a0d4b996e-kube-api-access-dc6tq\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.652365 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.652353    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571064f9-1393-43e1-a943-dd1a0d4b996e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.693696 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.693673    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 20:05:28.753189 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.753156    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571064f9-1393-43e1-a943-dd1a0d4b996e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.753330 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.753235    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.753330 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.753282    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.753483 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.753336    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc6tq\" (UniqueName: \"kubernetes.io/projected/571064f9-1393-43e1-a943-dd1a0d4b996e-kube-api-access-dc6tq\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.753572 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:05:28.753530    5551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-serving-cert: secret "maas-api-e2e-worker-single-main-42030c-serving-cert" not found
Oct 31 20:05:28.753638 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:05:28.753608    5551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls podName:571064f9-1393-43e1-a943-dd1a0d4b996e nodeName:}" failed. No retries permitted until 2026-10-31 20:05:29.253585081 +0000 UTC m=+6232.484823369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls") pod "maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq" (UID: "571064f9-1393-43e1-a943-dd1a0d4b996e") : secret "maas-api-e2e-worker-single-main-42030c-serving-cert" not found
Oct 31 20:05:28.753957 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.753938    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571064f9-1393-43e1-a943-dd1a0d4b996e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.755765 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.755748    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:28.811152 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:28.811068    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc6tq\" (UniqueName: \"kubernetes.io/projected/571064f9-1393-43e1-a943-dd1a0d4b996e-kube-api-access-dc6tq\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:29.258891 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:29.258840    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:29.261386 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:29.261359    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:29.454627 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:29.454582    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:29.834220 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:29.834193    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"]
Oct 31 20:05:29.836665 ip-10-0-132-99 kubenswrapper[5551]: W1031 20:05:29.836634    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571064f9_1393_43e1_a943_dd1a0d4b996e.slice/crio-4c5ece717d3d54a520a8fe85ed11865a809a9b23dd96695df6a6541640dae1c5 WatchSource:0}: Error finding container 4c5ece717d3d54a520a8fe85ed11865a809a9b23dd96695df6a6541640dae1c5: Status 404 returned error can't find the container with id 4c5ece717d3d54a520a8fe85ed11865a809a9b23dd96695df6a6541640dae1c5
Oct 31 20:05:29.838165 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:29.838147    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 20:05:30.133546 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:30.133514    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq" event={"ID":"571064f9-1393-43e1-a943-dd1a0d4b996e","Type":"ContainerStarted","Data":"4c5ece717d3d54a520a8fe85ed11865a809a9b23dd96695df6a6541640dae1c5"}
Oct 31 20:05:31.140271 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:31.140233    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq" event={"ID":"571064f9-1393-43e1-a943-dd1a0d4b996e","Type":"ContainerStarted","Data":"f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555"}
Oct 31 20:05:31.140717 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:31.140360    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:31.270758 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:31.270688    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq" podStartSLOduration=2.406449407 podStartE2EDuration="3.27066995s" totalImagesPullingTime="864.220543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:31.268848287 +0000 UTC m=+6234.500086590" watchObservedRunningTime="2026-10-31 20:05:31.27066995 +0000 UTC m=+6234.501908238"
Oct 31 20:05:37.152764 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:37.152734    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:05:46.603646 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:05:46.603604    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:05:51.853510 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.853472    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"]
Oct 31 20:05:51.858593 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.858564    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:51.904088 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.904058    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tls-certs\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:51.904246 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.904119    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-dshm\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:51.904301 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.904279    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-model-cache\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:51.904356 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.904338    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tmp-dir\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:51.904394 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.904373    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-home\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:51.904429 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.904412    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5gw9\" (UniqueName: \"kubernetes.io/projected/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-kube-api-access-n5gw9\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:51.911279 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.911239    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-42030c\"/\"openshift-service-ca.crt\""
Oct 31 20:05:51.911390 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.911310    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-42030c\"/\"kube-root-ca.crt\""
Oct 31 20:05:51.960646 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.960621    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-42030c\"/\"simulator-single-main-42030c-kserve-self-signed-certs\""
Oct 31 20:05:51.960804 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.960717    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-42030c\"/\"default-dockercfg-gf9qs\""
Oct 31 20:05:51.963150 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:51.963126    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"]
Oct 31 20:05:52.005979 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.005946    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-dshm\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.006210 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.006000    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-model-cache\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.006210 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.006102    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tmp-dir\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.006540 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.006505    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-model-cache\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.006540 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.006521    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tmp-dir\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.006756 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.006626    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-home\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.006756 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.006682    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5gw9\" (UniqueName: \"kubernetes.io/projected/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-kube-api-access-n5gw9\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.007000 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.006782    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tls-certs\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.008034 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.007005    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-home\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.010241 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.010194    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tls-certs\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.010768 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.010712    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-dshm\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.064542 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.064506    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5gw9\" (UniqueName: \"kubernetes.io/projected/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-kube-api-access-n5gw9\") pod \"simulator-single-main-42030c-kserve-79df5f5996-xp76t\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") " pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.172013 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.171965    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:05:52.552732 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:52.552706    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"]
Oct 31 20:05:52.554429 ip-10-0-132-99 kubenswrapper[5551]: W1031 20:05:52.554404    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bac64d1_6a0b_4f86_8948_cac0fc3c5267.slice/crio-76c6455912a1e6716124066c2fa6fc77772acb38ece6b26b0b888ebac3de44a1 WatchSource:0}: Error finding container 76c6455912a1e6716124066c2fa6fc77772acb38ece6b26b0b888ebac3de44a1: Status 404 returned error can't find the container with id 76c6455912a1e6716124066c2fa6fc77772acb38ece6b26b0b888ebac3de44a1
Oct 31 20:05:53.269974 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:53.269934    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t" event={"ID":"6bac64d1-6a0b-4f86-8948-cac0fc3c5267","Type":"ContainerStarted","Data":"42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946"}
Oct 31 20:05:53.269974 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:53.269974    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t" event={"ID":"6bac64d1-6a0b-4f86-8948-cac0fc3c5267","Type":"ContainerStarted","Data":"76c6455912a1e6716124066c2fa6fc77772acb38ece6b26b0b888ebac3de44a1"}
Oct 31 20:05:53.389957 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:53.389893    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t" podStartSLOduration=2.389875468 podStartE2EDuration="2.389875468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:53.388196415 +0000 UTC m=+6256.619434704" watchObservedRunningTime="2026-10-31 20:05:53.389875468 +0000 UTC m=+6256.621113759"
Oct 31 20:05:54.172916 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:05:54.172877    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:06:02.172153 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:02.172112    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:06:02.185247 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:02.185221    5551 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:06:02.336951 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:02.336910    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:06:16.603795 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:06:16.603755    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:06:41.313448 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:41.313420    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 20:06:41.372582 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:41.372546    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 20:06:41.380266 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:41.380243    5551 scope.go:122] "RemoveContainer" containerID="af8f628232a3f195a300ee47dedd062608bd0a494e9d7f0cb5d6741cd51a8bf3"
Oct 31 20:06:41.391146 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:41.391131    5551 scope.go:122] "RemoveContainer" containerID="edcd5a272307c9920c3abe764551288937c62268a48a79f3a99266e07ad34e65"
Oct 31 20:06:46.604099 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:06:46.604015    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:06:52.150263 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:52.150229    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"]
Oct 31 20:06:55.330048 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:55.330008    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"]
Oct 31 20:06:55.330564 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:55.330242    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq" podUID="571064f9-1393-43e1-a943-dd1a0d4b996e" containerName="maas-api" containerID="cri-o://f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555" gracePeriod=30
Oct 31 20:06:56.591097 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.591067    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:06:56.600378 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.600342    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-metrics-certs\") pod \"571064f9-1393-43e1-a943-dd1a0d4b996e\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") "
Oct 31 20:06:56.600537 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.600469    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls\") pod \"571064f9-1393-43e1-a943-dd1a0d4b996e\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") "
Oct 31 20:06:56.600537 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.600501    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571064f9-1393-43e1-a943-dd1a0d4b996e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571064f9-1393-43e1-a943-dd1a0d4b996e-openshift-service-ca\") pod \"571064f9-1393-43e1-a943-dd1a0d4b996e\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") "
Oct 31 20:06:56.600537 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.600525    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571064f9-1393-43e1-a943-dd1a0d4b996e-kube-api-access-dc6tq\" (UniqueName: \"kubernetes.io/projected/571064f9-1393-43e1-a943-dd1a0d4b996e-kube-api-access-dc6tq\") pod \"571064f9-1393-43e1-a943-dd1a0d4b996e\" (UID: \"571064f9-1393-43e1-a943-dd1a0d4b996e\") "
Oct 31 20:06:56.600946 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.600912    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571064f9-1393-43e1-a943-dd1a0d4b996e-openshift-service-ca" pod "571064f9-1393-43e1-a943-dd1a0d4b996e" (UID: "571064f9-1393-43e1-a943-dd1a0d4b996e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:56.602851 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.602823    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls" pod "571064f9-1393-43e1-a943-dd1a0d4b996e" (UID: "571064f9-1393-43e1-a943-dd1a0d4b996e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:56.602851 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.602839    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-metrics-certs" pod "571064f9-1393-43e1-a943-dd1a0d4b996e" (UID: "571064f9-1393-43e1-a943-dd1a0d4b996e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:56.602993 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.602851    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571064f9-1393-43e1-a943-dd1a0d4b996e-kube-api-access-dc6tq" pod "571064f9-1393-43e1-a943-dd1a0d4b996e" (UID: "571064f9-1393-43e1-a943-dd1a0d4b996e"). InnerVolumeSpecName "kube-api-access-dc6tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:56.645626 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.645542    5551 generic.go:356] "Generic (PLEG): container finished" podID="571064f9-1393-43e1-a943-dd1a0d4b996e" containerID="f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555" exitCode=0
Oct 31 20:06:56.645626 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.645592    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"
Oct 31 20:06:56.645842 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.645629    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq" event={"ID":"571064f9-1393-43e1-a943-dd1a0d4b996e","Type":"ContainerDied","Data":"f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555"}
Oct 31 20:06:56.645842 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.645685    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq" event={"ID":"571064f9-1393-43e1-a943-dd1a0d4b996e","Type":"ContainerDied","Data":"4c5ece717d3d54a520a8fe85ed11865a809a9b23dd96695df6a6541640dae1c5"}
Oct 31 20:06:56.645842 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.645714    5551 scope.go:122] "RemoveContainer" containerID="f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555"
Oct 31 20:06:56.656416 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.656394    5551 scope.go:122] "RemoveContainer" containerID="f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555"
Oct 31 20:06:56.656710 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:06:56.656694    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555\": container with ID starting with f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555 not found: ID does not exist" containerID="f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555"
Oct 31 20:06:56.656765 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.656719    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555"} err="failed to get container status \"f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555\": rpc error: code = NotFound desc = could not find container \"f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555\": container with ID starting with f01b22b1da24d73d0b01355703b42e43abfb952dae3304dc0bc0c141ec526555 not found: ID does not exist"
Oct 31 20:06:56.702285 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.702244    5551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-metrics-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:06:56.702285 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.702274    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571064f9-1393-43e1-a943-dd1a0d4b996e-maas-api-tls\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:06:56.702285 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.702285    5551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571064f9-1393-43e1-a943-dd1a0d4b996e-openshift-service-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:06:56.702550 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.702294    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc6tq\" (UniqueName: \"kubernetes.io/projected/571064f9-1393-43e1-a943-dd1a0d4b996e-kube-api-access-dc6tq\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:06:56.820542 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.820510    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"]
Oct 31 20:06:56.824782 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.824756    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42030c-66788887dc-ltjnq"]
Oct 31 20:06:56.909231 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:06:56.909144    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571064f9-1393-43e1-a943-dd1a0d4b996e" path="/var/lib/kubelet/pods/571064f9-1393-43e1-a943-dd1a0d4b996e/volumes"
Oct 31 20:07:07.172296 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.172237    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t" podUID="6bac64d1-6a0b-4f86-8948-cac0fc3c5267" containerName="main" containerID="cri-o://42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946" gracePeriod=45
Oct 31 20:07:07.325964 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.325921    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t" podUID="6bac64d1-6a0b-4f86-8948-cac0fc3c5267" containerName="main" probeResult="failure" output="Get \"https://10.134.0.252:8000/ready\": dial tcp 10.134.0.252:8000: connect: connection refused"
Oct 31 20:07:07.426150 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.426081    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:07:07.513031 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513000    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-dshm\") pod \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") "
Oct 31 20:07:07.513233 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513061    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tmp-dir\") pod \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") "
Oct 31 20:07:07.513233 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513104    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tls-certs\" (UniqueName: \"kubernetes.io/secret/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tls-certs\") pod \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") "
Oct 31 20:07:07.513233 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513127    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-model-cache\") pod \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") "
Oct 31 20:07:07.513233 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513145    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-home\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-home\") pod \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") "
Oct 31 20:07:07.513519 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513260    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-kube-api-access-n5gw9\" (UniqueName: \"kubernetes.io/projected/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-kube-api-access-n5gw9\") pod \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\" (UID: \"6bac64d1-6a0b-4f86-8948-cac0fc3c5267\") "
Oct 31 20:07:07.513519 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513390    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tmp-dir" pod "6bac64d1-6a0b-4f86-8948-cac0fc3c5267" (UID: "6bac64d1-6a0b-4f86-8948-cac0fc3c5267"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:07.513519 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513436    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-model-cache" pod "6bac64d1-6a0b-4f86-8948-cac0fc3c5267" (UID: "6bac64d1-6a0b-4f86-8948-cac0fc3c5267"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:07.513519 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513497    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-home" pod "6bac64d1-6a0b-4f86-8948-cac0fc3c5267" (UID: "6bac64d1-6a0b-4f86-8948-cac0fc3c5267"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:07.513746 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513642    5551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tmp-dir\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:07.513746 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513654    5551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-model-cache\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:07.513746 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.513664    5551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-home\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:07.515487 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.515445    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-dshm" pod "6bac64d1-6a0b-4f86-8948-cac0fc3c5267" (UID: "6bac64d1-6a0b-4f86-8948-cac0fc3c5267"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:07.515607 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.515532    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tls-certs" pod "6bac64d1-6a0b-4f86-8948-cac0fc3c5267" (UID: "6bac64d1-6a0b-4f86-8948-cac0fc3c5267"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:07.515607 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.515566    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-kube-api-access-n5gw9" pod "6bac64d1-6a0b-4f86-8948-cac0fc3c5267" (UID: "6bac64d1-6a0b-4f86-8948-cac0fc3c5267"). InnerVolumeSpecName "kube-api-access-n5gw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:07.615231 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.615198    5551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-tls-certs\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:07.615231 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.615228    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5gw9\" (UniqueName: \"kubernetes.io/projected/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-kube-api-access-n5gw9\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:07.615231 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.615239    5551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bac64d1-6a0b-4f86-8948-cac0fc3c5267-dshm\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:07.710905 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.710816    5551 generic.go:356] "Generic (PLEG): container finished" podID="6bac64d1-6a0b-4f86-8948-cac0fc3c5267" containerID="42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946" exitCode=0
Oct 31 20:07:07.710905 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.710868    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"
Oct 31 20:07:07.711126 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.710899    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t" event={"ID":"6bac64d1-6a0b-4f86-8948-cac0fc3c5267","Type":"ContainerDied","Data":"42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946"}
Oct 31 20:07:07.711126 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.710951    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t" event={"ID":"6bac64d1-6a0b-4f86-8948-cac0fc3c5267","Type":"ContainerDied","Data":"76c6455912a1e6716124066c2fa6fc77772acb38ece6b26b0b888ebac3de44a1"}
Oct 31 20:07:07.711126 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.710969    5551 scope.go:122] "RemoveContainer" containerID="42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946"
Oct 31 20:07:07.721228 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.721205    5551 scope.go:122] "RemoveContainer" containerID="42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946"
Oct 31 20:07:07.721676 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:07:07.721656    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946\": container with ID starting with 42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946 not found: ID does not exist" containerID="42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946"
Oct 31 20:07:07.721733 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.721688    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946"} err="failed to get container status \"42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946\": rpc error: code = NotFound desc = could not find container \"42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946\": container with ID starting with 42dda438900f461f96427c7fed6c27741f8b70c7f4c6b2c47aa5e0de8fe75946 not found: ID does not exist"
Oct 31 20:07:07.874196 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.874158    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"]
Oct 31 20:07:07.877926 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:07.877901    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-42030c/simulator-single-main-42030c-kserve-79df5f5996-xp76t"]
Oct 31 20:07:08.907924 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:08.907886    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bac64d1-6a0b-4f86-8948-cac0fc3c5267" path="/var/lib/kubelet/pods/6bac64d1-6a0b-4f86-8948-cac0fc3c5267/volumes"
Oct 31 20:07:16.604153 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:07:16.604112    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:07:33.920748 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.920702    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"]
Oct 31 20:07:33.921414 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.921394    5551 cpu_manager.go:414] "containerMap: removing container" podUID="6bac64d1-6a0b-4f86-8948-cac0fc3c5267" containerName="main"
Oct 31 20:07:33.921531 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.921416    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bac64d1-6a0b-4f86-8948-cac0fc3c5267" containerName="main"
Oct 31 20:07:33.921531 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.921436    5551 cpu_manager.go:414] "containerMap: removing container" podUID="571064f9-1393-43e1-a943-dd1a0d4b996e" containerName="maas-api"
Oct 31 20:07:33.921531 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.921444    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571064f9-1393-43e1-a943-dd1a0d4b996e" containerName="maas-api"
Oct 31 20:07:33.921715 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.921576    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bac64d1-6a0b-4f86-8948-cac0fc3c5267" containerName="main"
Oct 31 20:07:33.921715 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.921591    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="571064f9-1393-43e1-a943-dd1a0d4b996e" containerName="maas-api"
Oct 31 20:07:33.924820 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.924799    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:33.981793 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.981768    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"]
Oct 31 20:07:33.992686 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.992650    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:33.992816 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.992700    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqkqx\" (UniqueName: \"kubernetes.io/projected/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-kube-api-access-hqkqx\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:33.992897 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:33.992876    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:34.027977 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.027955    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:07:34.028063 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.027982    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:07:34.028063 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.027982    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:07:34.028063 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.028030    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:07:34.094109 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.094088    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:34.094226 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.094116    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkqx\" (UniqueName: \"kubernetes.io/projected/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-kube-api-access-hqkqx\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:34.094226 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.094180    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:34.094969 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.094887    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:34.096587 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.096568    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:34.152026 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.151988    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqkqx\" (UniqueName: \"kubernetes.io/projected/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-kube-api-access-hqkqx\") pod \"maas-api-revoke-sub-3d4156f1-tlc9c\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:34.237645 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.237562    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:34.613093 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.613065    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"]
Oct 31 20:07:34.615190 ip-10-0-132-99 kubenswrapper[5551]: W1031 20:07:34.615162    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd78829_92d0_4b6e_87c0_35d8ae7c3336.slice/crio-647377a7040c57e21a76cebb7ba372ab4234d6ab5fadea8047c91ef84d1641d8 WatchSource:0}: Error finding container 647377a7040c57e21a76cebb7ba372ab4234d6ab5fadea8047c91ef84d1641d8: Status 404 returned error can't find the container with id 647377a7040c57e21a76cebb7ba372ab4234d6ab5fadea8047c91ef84d1641d8
Oct 31 20:07:34.878547 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.878515    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c" event={"ID":"dfd78829-92d0-4b6e-87c0-35d8ae7c3336","Type":"ContainerStarted","Data":"e1ce0250a0c984e2cba696a4679e8cd3b9bcf72b001187e3408c10722215a6f5"}
Oct 31 20:07:34.878718 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.878577    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c" event={"ID":"dfd78829-92d0-4b6e-87c0-35d8ae7c3336","Type":"ContainerStarted","Data":"647377a7040c57e21a76cebb7ba372ab4234d6ab5fadea8047c91ef84d1641d8"}
Oct 31 20:07:34.992358 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:34.992307    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c" podStartSLOduration=1.9922934429999999 podStartE2EDuration="1.992293443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:07:34.990572589 +0000 UTC m=+6358.221810878" watchObservedRunningTime="2026-10-31 20:07:34.992293443 +0000 UTC m=+6358.223531733"
Oct 31 20:07:35.887218 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:35.887180    5551 generic.go:356] "Generic (PLEG): container finished" podID="dfd78829-92d0-4b6e-87c0-35d8ae7c3336" containerID="e1ce0250a0c984e2cba696a4679e8cd3b9bcf72b001187e3408c10722215a6f5" exitCode=0
Oct 31 20:07:35.887423 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:35.887275    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c" event={"ID":"dfd78829-92d0-4b6e-87c0-35d8ae7c3336","Type":"ContainerDied","Data":"e1ce0250a0c984e2cba696a4679e8cd3b9bcf72b001187e3408c10722215a6f5"}
Oct 31 20:07:37.036421 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.036398    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:37.125699 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.125675    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-openshift-ca\") pod \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") "
Oct 31 20:07:37.125835 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.125717    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-xks-ca\") pod \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") "
Oct 31 20:07:37.125835 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.125745    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-kube-api-access-hqkqx\" (UniqueName: \"kubernetes.io/projected/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-kube-api-access-hqkqx\") pod \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\" (UID: \"dfd78829-92d0-4b6e-87c0-35d8ae7c3336\") "
Oct 31 20:07:37.126034 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.126008    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-openshift-ca" pod "dfd78829-92d0-4b6e-87c0-35d8ae7c3336" (UID: "dfd78829-92d0-4b6e-87c0-35d8ae7c3336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:37.127897 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.127849    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-kube-api-access-hqkqx" pod "dfd78829-92d0-4b6e-87c0-35d8ae7c3336" (UID: "dfd78829-92d0-4b6e-87c0-35d8ae7c3336"). InnerVolumeSpecName "kube-api-access-hqkqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:37.128037 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.127896    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-xks-ca" pod "dfd78829-92d0-4b6e-87c0-35d8ae7c3336" (UID: "dfd78829-92d0-4b6e-87c0-35d8ae7c3336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:37.226922 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.226833    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-openshift-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.226922 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.226877    5551 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-maas-api-xks-ca\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.226922 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.226890    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqkqx\" (UniqueName: \"kubernetes.io/projected/dfd78829-92d0-4b6e-87c0-35d8ae7c3336-kube-api-access-hqkqx\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.898935 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.898902    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"
Oct 31 20:07:37.898935 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.898918    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c" event={"ID":"dfd78829-92d0-4b6e-87c0-35d8ae7c3336","Type":"ContainerDied","Data":"647377a7040c57e21a76cebb7ba372ab4234d6ab5fadea8047c91ef84d1641d8"}
Oct 31 20:07:37.899156 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:37.898970    5551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647377a7040c57e21a76cebb7ba372ab4234d6ab5fadea8047c91ef84d1641d8"
Oct 31 20:07:46.604366 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:07:46.604326    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:07:52.774718 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:52.774679    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"]
Oct 31 20:07:52.775214 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:52.774982    5551 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr" podUID="e8c99ace-8a40-41aa-9615-809def49b98f" containerName="manager" containerID="cri-o://74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3" gracePeriod=10
Oct 31 20:07:54.239570 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:54.239534    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 20:07:54.344599 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:54.344512    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c99ace-8a40-41aa-9615-809def49b98f-kube-api-access-q7l2g\" (UniqueName: \"kubernetes.io/projected/e8c99ace-8a40-41aa-9615-809def49b98f-kube-api-access-q7l2g\") pod \"e8c99ace-8a40-41aa-9615-809def49b98f\" (UID: \"e8c99ace-8a40-41aa-9615-809def49b98f\") "
Oct 31 20:07:54.344844 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:54.344649    5551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c99ace-8a40-41aa-9615-809def49b98f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8c99ace-8a40-41aa-9615-809def49b98f-extensions-socket-volume\") pod \"e8c99ace-8a40-41aa-9615-809def49b98f\" (UID: \"e8c99ace-8a40-41aa-9615-809def49b98f\") "
Oct 31 20:07:54.345040 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:54.345017    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c99ace-8a40-41aa-9615-809def49b98f-extensions-socket-volume" pod "e8c99ace-8a40-41aa-9615-809def49b98f" (UID: "e8c99ace-8a40-41aa-9615-809def49b98f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:54.346771 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:54.346746    5551 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c99ace-8a40-41aa-9615-809def49b98f-kube-api-access-q7l2g" pod "e8c99ace-8a40-41aa-9615-809def49b98f" (UID: "e8c99ace-8a40-41aa-9615-809def49b98f"). InnerVolumeSpecName "kube-api-access-q7l2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:54.445889 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:54.445860    5551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7l2g\" (UniqueName: \"kubernetes.io/projected/e8c99ace-8a40-41aa-9615-809def49b98f-kube-api-access-q7l2g\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:54.445889 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:54.445885    5551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8c99ace-8a40-41aa-9615-809def49b98f-extensions-socket-volume\") on node \"ip-10-0-132-99.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:55.021774 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.021734    5551 generic.go:356] "Generic (PLEG): container finished" podID="e8c99ace-8a40-41aa-9615-809def49b98f" containerID="74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3" exitCode=0
Oct 31 20:07:55.021964 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.021779    5551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"
Oct 31 20:07:55.021964 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.021809    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr" event={"ID":"e8c99ace-8a40-41aa-9615-809def49b98f","Type":"ContainerDied","Data":"74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3"}
Oct 31 20:07:55.021964 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.021855    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr" event={"ID":"e8c99ace-8a40-41aa-9615-809def49b98f","Type":"ContainerDied","Data":"0316e25ae5bd4ee23db657767f8d1c9244bbf3bb4aeb927ca987cdfdb0aed27c"}
Oct 31 20:07:55.021964 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.021881    5551 scope.go:122] "RemoveContainer" containerID="74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3"
Oct 31 20:07:55.038393 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.038372    5551 scope.go:122] "RemoveContainer" containerID="74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3"
Oct 31 20:07:55.038698 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:07:55.038673    5551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3\": container with ID starting with 74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3 not found: ID does not exist" containerID="74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3"
Oct 31 20:07:55.038774 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.038711    5551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3"} err="failed to get container status \"74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3\": rpc error: code = NotFound desc = could not find container \"74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3\": container with ID starting with 74bb4a2d85b82072a5d86ad90083df32204cb56c8284b549e6645c749c54b1b3 not found: ID does not exist"
Oct 31 20:07:55.248039 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.247962    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"]
Oct 31 20:07:55.296671 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:55.296641    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q2zdr"]
Oct 31 20:07:56.908645 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:07:56.908614    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c99ace-8a40-41aa-9615-809def49b98f" path="/var/lib/kubelet/pods/e8c99ace-8a40-41aa-9615-809def49b98f/volumes"
Oct 31 20:08:16.603195 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:08:16.603150    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:08:46.603733 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:08:46.603687    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:08:56.350944 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.350904    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"]
Oct 31 20:08:56.351430 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.351342    5551 cpu_manager.go:414] "containerMap: removing container" podUID="e8c99ace-8a40-41aa-9615-809def49b98f" containerName="manager"
Oct 31 20:08:56.351430 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.351354    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c99ace-8a40-41aa-9615-809def49b98f" containerName="manager"
Oct 31 20:08:56.351430 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.351380    5551 cpu_manager.go:414] "containerMap: removing container" podUID="dfd78829-92d0-4b6e-87c0-35d8ae7c3336" containerName="revoke-keys"
Oct 31 20:08:56.351430 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.351385    5551 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd78829-92d0-4b6e-87c0-35d8ae7c3336" containerName="revoke-keys"
Oct 31 20:08:56.351621 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.351445    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c99ace-8a40-41aa-9615-809def49b98f" containerName="manager"
Oct 31 20:08:56.351621 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.351473    5551 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd78829-92d0-4b6e-87c0-35d8ae7c3336" containerName="revoke-keys"
Oct 31 20:08:56.354639 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.354622    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:56.457361 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.457329    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6tbfg\""
Oct 31 20:08:56.459597 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.459575    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"]
Oct 31 20:08:56.464827 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.464801    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e3656ae-5862-4c6c-8ccd-a46da73fb260-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b78q8\" (UID: \"0e3656ae-5862-4c6c-8ccd-a46da73fb260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:56.464939 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.464830    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz29w\" (UniqueName: \"kubernetes.io/projected/0e3656ae-5862-4c6c-8ccd-a46da73fb260-kube-api-access-rz29w\") pod \"kuadrant-operator-controller-manager-678d87766f-b78q8\" (UID: \"0e3656ae-5862-4c6c-8ccd-a46da73fb260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:56.566385 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.566344    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e3656ae-5862-4c6c-8ccd-a46da73fb260-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b78q8\" (UID: \"0e3656ae-5862-4c6c-8ccd-a46da73fb260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:56.566385 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.566385    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz29w\" (UniqueName: \"kubernetes.io/projected/0e3656ae-5862-4c6c-8ccd-a46da73fb260-kube-api-access-rz29w\") pod \"kuadrant-operator-controller-manager-678d87766f-b78q8\" (UID: \"0e3656ae-5862-4c6c-8ccd-a46da73fb260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:56.566759 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.566736    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e3656ae-5862-4c6c-8ccd-a46da73fb260-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b78q8\" (UID: \"0e3656ae-5862-4c6c-8ccd-a46da73fb260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:56.626883 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.626859    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz29w\" (UniqueName: \"kubernetes.io/projected/0e3656ae-5862-4c6c-8ccd-a46da73fb260-kube-api-access-rz29w\") pod \"kuadrant-operator-controller-manager-678d87766f-b78q8\" (UID: \"0e3656ae-5862-4c6c-8ccd-a46da73fb260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:56.667961 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:56.667934    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:57.148413 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:57.148377    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"]
Oct 31 20:08:57.200860 ip-10-0-132-99 kubenswrapper[5551]: W1031 20:08:57.200827    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3656ae_5862_4c6c_8ccd_a46da73fb260.slice/crio-93caf2499c66f158aafaf1bd3ed55c1b76a21cac6fbf8201b8b5c2dc7511b543 WatchSource:0}: Error finding container 93caf2499c66f158aafaf1bd3ed55c1b76a21cac6fbf8201b8b5c2dc7511b543: Status 404 returned error can't find the container with id 93caf2499c66f158aafaf1bd3ed55c1b76a21cac6fbf8201b8b5c2dc7511b543
Oct 31 20:08:57.377424 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:57.377391    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8" event={"ID":"0e3656ae-5862-4c6c-8ccd-a46da73fb260","Type":"ContainerStarted","Data":"398fe1d76918205b24d4d20247b25ddf91ada3cad8052ad870db49fa7fe0e9db"}
Oct 31 20:08:57.377860 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:57.377450    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8" event={"ID":"0e3656ae-5862-4c6c-8ccd-a46da73fb260","Type":"ContainerStarted","Data":"93caf2499c66f158aafaf1bd3ed55c1b76a21cac6fbf8201b8b5c2dc7511b543"}
Oct 31 20:08:57.377860 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:57.377505    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:08:57.498218 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:08:57.498116    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8" podStartSLOduration=1.498101001 podStartE2EDuration="1.498101001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:57.497236601 +0000 UTC m=+6440.728474892" watchObservedRunningTime="2026-10-31 20:08:57.498101001 +0000 UTC m=+6440.729339291"
Oct 31 20:09:08.384727 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:08.384695    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b78q8"
Oct 31 20:09:16.606338 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:09:16.606231    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:09:46.605019 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:09:46.604934    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:09:55.256671 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.256635    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68474f84db-b69mr"]
Oct 31 20:09:55.260573 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.260552    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.314048 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.314009    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gj6\" (UniqueName: \"kubernetes.io/projected/f133c50a-46df-4607-908a-1d4f553e77af-kube-api-access-z7gj6\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.314245 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.314061    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f133c50a-46df-4607-908a-1d4f553e77af-metrics-certs\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.314245 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.314085    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f133c50a-46df-4607-908a-1d4f553e77af-maas-api-tls\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.314367 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.314250    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f133c50a-46df-4607-908a-1d4f553e77af-openshift-service-ca\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.324210 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.324186    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:09:55.367363 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.367336    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:09:55.371210 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.371184    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68474f84db-b69mr"]
Oct 31 20:09:55.393476 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.393433    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 20:09:55.393618 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.393506    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 20:09:55.415495 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.415450    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f133c50a-46df-4607-908a-1d4f553e77af-openshift-service-ca\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.415625 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.415565    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gj6\" (UniqueName: \"kubernetes.io/projected/f133c50a-46df-4607-908a-1d4f553e77af-kube-api-access-z7gj6\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.415625 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.415595    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f133c50a-46df-4607-908a-1d4f553e77af-metrics-certs\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.415625 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.415612    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f133c50a-46df-4607-908a-1d4f553e77af-maas-api-tls\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.416117 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.416094    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f133c50a-46df-4607-908a-1d4f553e77af-openshift-service-ca\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.416735 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.416709    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 20:09:55.418197 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.418176    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f133c50a-46df-4607-908a-1d4f553e77af-maas-api-tls\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.428198 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.428175    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f133c50a-46df-4607-908a-1d4f553e77af-metrics-certs\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.480061 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.480040    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gj6\" (UniqueName: \"kubernetes.io/projected/f133c50a-46df-4607-908a-1d4f553e77af-kube-api-access-z7gj6\") pod \"maas-api-68474f84db-b69mr\" (UID: \"f133c50a-46df-4607-908a-1d4f553e77af\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.575449 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.575384    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:55.951636 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:55.951611    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68474f84db-b69mr"]
Oct 31 20:09:55.953361 ip-10-0-132-99 kubenswrapper[5551]: W1031 20:09:55.953338    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf133c50a_46df_4607_908a_1d4f553e77af.slice/crio-3cb1b99443a245c6f1a43b8aaaccf450de8bf45e66c585a5b8ad32aaba353759 WatchSource:0}: Error finding container 3cb1b99443a245c6f1a43b8aaaccf450de8bf45e66c585a5b8ad32aaba353759: Status 404 returned error can't find the container with id 3cb1b99443a245c6f1a43b8aaaccf450de8bf45e66c585a5b8ad32aaba353759
Oct 31 20:09:56.810331 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:56.810295    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr" event={"ID":"f133c50a-46df-4607-908a-1d4f553e77af","Type":"ContainerStarted","Data":"3cb1b99443a245c6f1a43b8aaaccf450de8bf45e66c585a5b8ad32aaba353759"}
Oct 31 20:09:57.817674 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:57.817629    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr" event={"ID":"f133c50a-46df-4607-908a-1d4f553e77af","Type":"ContainerStarted","Data":"4e02ee0d9435c65596ca0bd73f5be9a2432a90bef38301267e537f207fa7a2f8"}
Oct 31 20:09:57.818081 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:57.817702    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:09:57.932176 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:09:57.932082    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr" podStartSLOduration=2.064415655 podStartE2EDuration="2.932069033s" totalImagesPullingTime="867.653378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:57.931695973 +0000 UTC m=+6501.162934263" watchObservedRunningTime="2026-10-31 20:09:57.932069033 +0000 UTC m=+6501.163307323"
Oct 31 20:10:03.828328 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:10:03.828297    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68474f84db-b69mr"
Oct 31 20:10:16.604218 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:10:16.604175    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:10:46.605272 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:10:46.605234    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:11:16.604029 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:11:16.603939    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:11:36.423598 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:11:36.423558    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-p9pnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:11:36.424293 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:11:36.424256    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-p9pnj" podUID="a7b0a99b-0d21-4232-a701-7f14c6d33748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:11:41.357877 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:11:41.357849    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 20:11:41.412637 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:11:41.412611    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 20:11:46.605507 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:11:46.605434    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:12:16.749744 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:12:16.749675    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:12:17.716616 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:12:17.716551    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:12:17.716994 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:12:17.716950    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:12:17.790599 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:12:17.790562    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:12:17.791213 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:12:17.791181    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:12:38.047106 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:12:38.047072    5551 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"]
Oct 31 20:12:38.096416 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:12:38.096389    5551 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tlc9c"]
Oct 31 20:12:38.910334 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:12:38.910293    5551 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd78829-92d0-4b6e-87c0-35d8ae7c3336" path="/var/lib/kubelet/pods/dfd78829-92d0-4b6e-87c0-35d8ae7c3336/volumes"
Oct 31 20:12:46.603869 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:12:46.603779    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:13:16.729493 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:13:16.716314    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:13:41.521584 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:13:41.521547    5551 scope.go:122] "RemoveContainer" containerID="e1ce0250a0c984e2cba696a4679e8cd3b9bcf72b001187e3408c10722215a6f5"
Oct 31 20:13:46.771731 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:13:46.771681    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:13:47.722627 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:13:47.722583    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:13:47.723086 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:13:47.723034    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:13:47.781663 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:13:47.781616    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:13:47.782285 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:13:47.782233    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:14:16.604299 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:14:16.604208    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:14:32.778484 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:14:32.777558    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:14:32.778484 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:14:32.777625    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:14:46.605258 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:14:46.605211    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:15:04.568355 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:15:04.568312    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mjzjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:15:04.569184 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:15:04.569145    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mjzjn" podUID="5d6cba1a-6e61-4893-a34e-0690f844e329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:15:16.607387 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:15:16.607350    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:15:46.784917 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:15:46.784691    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:15:47.727657 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:15:47.727609    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-8z84d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:15:47.727992 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:15:47.727953    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-8z84d" podUID="e64f2c2f-b471-4e6e-a597-a11a9975a428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:15:47.793272 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:15:47.793225    5551 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-b8w4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:15:47.793978 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:15:47.793927    5551 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-b8w4r" podUID="75e922ca-483e-4f40-b4c0-c47f5fa1a9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:16:16.654675 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:16:16.654622    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:16:25.366048 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:25.365995    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d67dfb554-c2w6p_e22ad853-ddd2-4669-8df9-3c3f67f659f2/authorino/0.log"
Oct 31 20:16:38.120618 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:38.120586    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sr4rh_58ef3323-2763-4091-8d61-f4600a97ad9b/manager/0.log"
Oct 31 20:16:41.358585 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:41.358537    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68474f84db-b69mr_f133c50a-46df-4607-908a-1d4f553e77af/maas-api/0.log"
Oct 31 20:16:41.409637 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:41.409612    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 20:16:41.451282 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:41.451256    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zzlkq_8205297a-7593-4712-b393-e1fabfff8a15/extract/0.log"
Oct 31 20:16:46.604375 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:16:46.604341    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:16:47.696419 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:47.696391    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d67dfb554-c2w6p_e22ad853-ddd2-4669-8df9-3c3f67f659f2/authorino/0.log"
Oct 31 20:16:48.079962 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:48.079885    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-sbwg7_14e622e5-332a-4304-a9ec-d0feaa7c4002/manager/0.log"
Oct 31 20:16:48.315120 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:48.315088    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zkmz5_99d1f84b-a604-46c8-83be-3f53ab8c959c/manager/0.log"
Oct 31 20:16:48.973347 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:48.973320    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-b78q8_0e3656ae-5862-4c6c-8ccd-a46da73fb260/manager/0.log"
Oct 31 20:16:49.457849 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:49.457817    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hd4jn_2ff429fc-0746-4d6a-aff4-088caec1919e/manager/1.log"
Oct 31 20:16:50.162558 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:50.162523    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hd4jn_2ff429fc-0746-4d6a-aff4-088caec1919e/manager/0.log"
Oct 31 20:16:50.791417 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:50.791385    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq_42ddf73a-f3d8-46de-b88f-a5d887e21df8/istio-proxy/0.log"
Oct 31 20:16:51.965878 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:51.965846    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-8z84d_e64f2c2f-b471-4e6e-a597-a11a9975a428/istio-proxy/0.log"
Oct 31 20:16:52.206744 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:52.206714    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-b8w4r_75e922ca-483e-4f40-b4c0-c47f5fa1a9cb/istio-proxy/0.log"
Oct 31 20:16:53.246092 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:53.246064    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mjzjn_5d6cba1a-6e61-4893-a34e-0690f844e329/istio-proxy/0.log"
Oct 31 20:16:53.468307 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:53.468274    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-p9pnj_a7b0a99b-0d21-4232-a701-7f14c6d33748/istio-proxy/0.log"
Oct 31 20:16:54.255673 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:54.255640    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls_3c19da20-58a9-4a4d-9fe4-7ea13270d595/istio-proxy/0.log"
Oct 31 20:16:56.563598 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:56.563564    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wj9dz_bd7eec1c-4593-41c1-8eca-cbf856f87ab5/main/0.log"
Oct 31 20:16:57.075469 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:57.075427    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89bmxr_8c0be12c-0277-48af-881a-0e0da6c70c2f/main/0.log"
Oct 31 20:16:57.345026 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:57.344950    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2whxh_2ed09c00-e8fa-4c44-a99c-aaf2d91a5153/main/0.log"
Oct 31 20:16:57.845928 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:16:57.845897    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-kgz86_35c185c9-da97-4e98-a6e3-36664eba8faf/main/0.log"
Oct 31 20:17:07.954768 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:07.954726    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fj44t/must-gather-9mspx"]
Oct 31 20:17:07.958598 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:07.958579    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fj44t/must-gather-9mspx"
Oct 31 20:17:08.034496 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.034447    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq8fl\" (UniqueName: \"kubernetes.io/projected/dd5165d0-c361-4da1-89b5-b90bdb38158e-kube-api-access-bq8fl\") pod \"must-gather-9mspx\" (UID: \"dd5165d0-c361-4da1-89b5-b90bdb38158e\") " pod="openshift-must-gather-fj44t/must-gather-9mspx"
Oct 31 20:17:08.034650 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.034576    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd5165d0-c361-4da1-89b5-b90bdb38158e-must-gather-output\") pod \"must-gather-9mspx\" (UID: \"dd5165d0-c361-4da1-89b5-b90bdb38158e\") " pod="openshift-must-gather-fj44t/must-gather-9mspx"
Oct 31 20:17:08.061725 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.061695    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fj44t\"/\"openshift-service-ca.crt\""
Oct 31 20:17:08.061888 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.061711    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fj44t\"/\"default-dockercfg-7cp4z\""
Oct 31 20:17:08.061888 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.061734    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fj44t\"/\"kube-root-ca.crt\""
Oct 31 20:17:08.065491 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.065448    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fj44t/must-gather-9mspx"]
Oct 31 20:17:08.135622 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.135586    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8fl\" (UniqueName: \"kubernetes.io/projected/dd5165d0-c361-4da1-89b5-b90bdb38158e-kube-api-access-bq8fl\") pod \"must-gather-9mspx\" (UID: \"dd5165d0-c361-4da1-89b5-b90bdb38158e\") " pod="openshift-must-gather-fj44t/must-gather-9mspx"
Oct 31 20:17:08.135763 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.135645    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd5165d0-c361-4da1-89b5-b90bdb38158e-must-gather-output\") pod \"must-gather-9mspx\" (UID: \"dd5165d0-c361-4da1-89b5-b90bdb38158e\") " pod="openshift-must-gather-fj44t/must-gather-9mspx"
Oct 31 20:17:08.136014 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.135994    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd5165d0-c361-4da1-89b5-b90bdb38158e-must-gather-output\") pod \"must-gather-9mspx\" (UID: \"dd5165d0-c361-4da1-89b5-b90bdb38158e\") " pod="openshift-must-gather-fj44t/must-gather-9mspx"
Oct 31 20:17:08.194093 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.194067    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8fl\" (UniqueName: \"kubernetes.io/projected/dd5165d0-c361-4da1-89b5-b90bdb38158e-kube-api-access-bq8fl\") pod \"must-gather-9mspx\" (UID: \"dd5165d0-c361-4da1-89b5-b90bdb38158e\") " pod="openshift-must-gather-fj44t/must-gather-9mspx"
Oct 31 20:17:08.273439 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.273377    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fj44t/must-gather-9mspx"
Oct 31 20:17:08.650297 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.650272    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fj44t/must-gather-9mspx"]
Oct 31 20:17:08.652029 ip-10-0-132-99 kubenswrapper[5551]: W1031 20:17:08.652000    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd5165d0_c361_4da1_89b5_b90bdb38158e.slice/crio-6dd6a3d967dbde1e540328362aa4567f1e9ff588a430ae649a98b6bd8c58e7a6 WatchSource:0}: Error finding container 6dd6a3d967dbde1e540328362aa4567f1e9ff588a430ae649a98b6bd8c58e7a6: Status 404 returned error can't find the container with id 6dd6a3d967dbde1e540328362aa4567f1e9ff588a430ae649a98b6bd8c58e7a6
Oct 31 20:17:08.654005 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:08.653982    5551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 20:17:09.275847 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:09.275807    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fj44t/must-gather-9mspx" event={"ID":"dd5165d0-c361-4da1-89b5-b90bdb38158e","Type":"ContainerStarted","Data":"6dd6a3d967dbde1e540328362aa4567f1e9ff588a430ae649a98b6bd8c58e7a6"}
Oct 31 20:17:16.604471 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:17:16.604362    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:17:17.470275 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:17.470239    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m56gf_28594e99-2b75-44a6-8922-315e6b680700/global-pull-secret-syncer/0.log"
Oct 31 20:17:20.585670 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:20.585634    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hgxrq_2efd8852-abb6-40d2-afda-40fe91bbd18f/konnectivity-agent/0.log"
Oct 31 20:17:21.390190 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:21.390144    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-99.us-west-1.compute.internal_401f601c1d615f3dbfc663ac1fbee6e2/haproxy/0.log"
Oct 31 20:17:22.358244 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:22.358203    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fj44t/must-gather-9mspx" event={"ID":"dd5165d0-c361-4da1-89b5-b90bdb38158e","Type":"ContainerStarted","Data":"d5585b053e821b73b3741dc557833154a4e00fe4719045880f2c8ff6df710574"}
Oct 31 20:17:22.358817 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:22.358258    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fj44t/must-gather-9mspx" event={"ID":"dd5165d0-c361-4da1-89b5-b90bdb38158e","Type":"ContainerStarted","Data":"842a9436f6e876edf662c50819dcad2a1a52f8b632f4844a6f89998e0221a361"}
Oct 31 20:17:22.478476 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:22.478408    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fj44t/must-gather-9mspx" podStartSLOduration=2.86393738 podStartE2EDuration="15.478391508s" totalImagesPullingTime="12.614454128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:17:22.475477842 +0000 UTC m=+6945.706716131" watchObservedRunningTime="2026-10-31 20:17:22.478391508 +0000 UTC m=+6945.709629802"
Oct 31 20:17:25.285388 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:25.285343    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d67dfb554-c2w6p_e22ad853-ddd2-4669-8df9-3c3f67f659f2/authorino/0.log"
Oct 31 20:17:26.104554 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.104526    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sr4rh_58ef3323-2763-4091-8d61-f4600a97ad9b/manager/0.log"
Oct 31 20:17:26.779935 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.779901    5551 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"]
Oct 31 20:17:26.785856 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.785830    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.794971 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.794952    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-sbwg7_14e622e5-332a-4304-a9ec-d0feaa7c4002/manager/0.log"
Oct 31 20:17:26.834795 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.834697    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-sys\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.834795 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.834736    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-podres\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.835035 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.834886    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-proc\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.835035 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.834996    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-lib-modules\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.835164 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.835057    5551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbss\" (UniqueName: \"kubernetes.io/projected/f181ee20-d63f-4b7d-8834-0ced9681502c-kube-api-access-dsbss\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.845705 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.845680    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"]
Oct 31 20:17:26.923642 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.923615    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bxfnk\"/\"openshift-service-ca.crt\""
Oct 31 20:17:26.923808 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.923662    5551 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bxfnk\"/\"kube-root-ca.crt\""
Oct 31 20:17:26.936082 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936053    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-sys\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.936164 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936089    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-podres\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.936164 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936157    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-proc\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.936259 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936167    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-sys\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.936259 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936245    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-lib-modules\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.936364 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936276    5551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbss\" (UniqueName: \"kubernetes.io/projected/f181ee20-d63f-4b7d-8834-0ced9681502c-kube-api-access-dsbss\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.936364 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936320    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-lib-modules\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.936364 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936276    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-proc\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.936364 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.936320    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f181ee20-d63f-4b7d-8834-0ced9681502c-podres\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:26.955904 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.955883    5551 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bxfnk\"/\"default-dockercfg-l2c59\""
Oct 31 20:17:26.993651 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:26.993622    5551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbss\" (UniqueName: \"kubernetes.io/projected/f181ee20-d63f-4b7d-8834-0ced9681502c-kube-api-access-dsbss\") pod \"perf-node-gather-daemonset-jdvvg\" (UID: \"f181ee20-d63f-4b7d-8834-0ced9681502c\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:27.099993 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:27.099922    5551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:27.217080 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:27.217025    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zkmz5_99d1f84b-a604-46c8-83be-3f53ab8c959c/manager/0.log"
Oct 31 20:17:27.477168 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:27.477141    5551 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"]
Oct 31 20:17:27.480062 ip-10-0-132-99 kubenswrapper[5551]: W1031 20:17:27.480035    5551 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf181ee20_d63f_4b7d_8834_0ced9681502c.slice/crio-e534374115b6d75a078a4d4f424cf9250eb33b54cd87527d92ec073b90217401 WatchSource:0}: Error finding container e534374115b6d75a078a4d4f424cf9250eb33b54cd87527d92ec073b90217401: Status 404 returned error can't find the container with id e534374115b6d75a078a4d4f424cf9250eb33b54cd87527d92ec073b90217401
Oct 31 20:17:28.166125 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:28.166098    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-b78q8_0e3656ae-5862-4c6c-8ccd-a46da73fb260/manager/0.log"
Oct 31 20:17:28.399661 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:28.399622    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg" event={"ID":"f181ee20-d63f-4b7d-8834-0ced9681502c","Type":"ContainerStarted","Data":"4925eed9aa305154f1258d74c890db83fe2da3a23e74843aff07777ee3d7b39c"}
Oct 31 20:17:28.399859 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:28.399711    5551 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg" event={"ID":"f181ee20-d63f-4b7d-8834-0ced9681502c","Type":"ContainerStarted","Data":"e534374115b6d75a078a4d4f424cf9250eb33b54cd87527d92ec073b90217401"}
Oct 31 20:17:28.399859 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:28.399785    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:28.529560 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:28.529442    5551 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg" podStartSLOduration=2.529426729 podStartE2EDuration="2.529426729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:17:28.52752334 +0000 UTC m=+6951.758761626" watchObservedRunningTime="2026-10-31 20:17:28.529426729 +0000 UTC m=+6951.760665010"
Oct 31 20:17:28.979509 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:28.979480    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hd4jn_2ff429fc-0746-4d6a-aff4-088caec1919e/manager/0.log"
Oct 31 20:17:29.034124 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:29.034093    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hd4jn_2ff429fc-0746-4d6a-aff4-088caec1919e/manager/1.log"
Oct 31 20:17:31.516900 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:31.516869    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_169ebe1f-facb-4fbb-96cb-2535f0634218/alertmanager/0.log"
Oct 31 20:17:32.059668 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:32.059640    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_169ebe1f-facb-4fbb-96cb-2535f0634218/config-reloader/0.log"
Oct 31 20:17:32.119384 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:32.119343    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sr4rh_58ef3323-2763-4091-8d61-f4600a97ad9b/manager/0.log"
Oct 31 20:17:32.485842 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:32.485788    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_169ebe1f-facb-4fbb-96cb-2535f0634218/kube-rbac-proxy-web/0.log"
Oct 31 20:17:32.921079 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:32.921050    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_169ebe1f-facb-4fbb-96cb-2535f0634218/kube-rbac-proxy/0.log"
Oct 31 20:17:33.335768 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:33.335671    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_169ebe1f-facb-4fbb-96cb-2535f0634218/kube-rbac-proxy-metric/0.log"
Oct 31 20:17:33.751421 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:33.751388    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_169ebe1f-facb-4fbb-96cb-2535f0634218/prom-label-proxy/0.log"
Oct 31 20:17:34.168185 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:34.168154    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_169ebe1f-facb-4fbb-96cb-2535f0634218/init-config-reloader/0.log"
Oct 31 20:17:34.416661 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:34.416630    5551 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-jdvvg"
Oct 31 20:17:35.009557 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:35.009529    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tdhb5_65c82ec5-b112-4e22-9e89-25b3a69fe6a2/kube-state-metrics/0.log"
Oct 31 20:17:35.419704 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:35.419676    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tdhb5_65c82ec5-b112-4e22-9e89-25b3a69fe6a2/kube-rbac-proxy-main/0.log"
Oct 31 20:17:35.837018 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:35.836911    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tdhb5_65c82ec5-b112-4e22-9e89-25b3a69fe6a2/kube-rbac-proxy-self/0.log"
Oct 31 20:17:36.256192 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:36.256161    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7ff57c786f-q298q_cb3b9ece-b002-4973-95ed-862cd33ca046/metrics-server/0.log"
Oct 31 20:17:37.834844 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:37.834814    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sr4rh_58ef3323-2763-4091-8d61-f4600a97ad9b/manager/0.log"
Oct 31 20:17:38.399779 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:38.399747    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fgkm2_83fe51c6-ee5f-4ccf-a934-6d42a27d475b/node-exporter/0.log"
Oct 31 20:17:38.818197 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:38.818122    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fgkm2_83fe51c6-ee5f-4ccf-a934-6d42a27d475b/kube-rbac-proxy/0.log"
Oct 31 20:17:39.236523 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:39.236494    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fgkm2_83fe51c6-ee5f-4ccf-a934-6d42a27d475b/init-textfile/0.log"
Oct 31 20:17:40.910728 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:40.910684    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2dktv_445c9865-e1dc-467f-bb6f-f29e8c6b7f68/kube-rbac-proxy-main/0.log"
Oct 31 20:17:41.324676 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:41.324564    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2dktv_445c9865-e1dc-467f-bb6f-f29e8c6b7f68/kube-rbac-proxy-self/0.log"
Oct 31 20:17:41.738437 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:41.738406    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2dktv_445c9865-e1dc-467f-bb6f-f29e8c6b7f68/openshift-state-metrics/0.log"
Oct 31 20:17:43.487771 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:43.487736    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq_42ddf73a-f3d8-46de-b88f-a5d887e21df8/istio-proxy/0.log"
Oct 31 20:17:45.088979 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:45.088952    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wzh2n_461b92ce-a67b-4dbe-924b-9e7c84f07a02/prometheus-operator/0.log"
Oct 31 20:17:45.521327 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:45.521293    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wzh2n_461b92ce-a67b-4dbe-924b-9e7c84f07a02/kube-rbac-proxy/0.log"
Oct 31 20:17:45.926637 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:45.926601    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-8z84d_e64f2c2f-b471-4e6e-a597-a11a9975a428/istio-proxy/0.log"
Oct 31 20:17:46.139359 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:46.139318    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-b8w4r_75e922ca-483e-4f40-b4c0-c47f5fa1a9cb/istio-proxy/0.log"
Oct 31 20:17:46.371261 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:46.371157    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76f4cd98b8-hrnz4_884384ea-01db-45f2-8c04-2163de1950ca/telemeter-client/0.log"
Oct 31 20:17:46.605854 ip-10-0-132-99 kubenswrapper[5551]: E1031 20:17:46.605807    5551 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g8rpt" containerName="extractor"
Oct 31 20:17:46.790433 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:46.790403    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76f4cd98b8-hrnz4_884384ea-01db-45f2-8c04-2163de1950ca/reload/0.log"
Oct 31 20:17:47.000904 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:47.000876    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mjzjn_5d6cba1a-6e61-4893-a34e-0690f844e329/istio-proxy/0.log"
Oct 31 20:17:47.207682 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:47.207652    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76f4cd98b8-hrnz4_884384ea-01db-45f2-8c04-2163de1950ca/kube-rbac-proxy/0.log"
Oct 31 20:17:47.213759 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:47.213738    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-p9pnj_a7b0a99b-0d21-4232-a701-7f14c6d33748/istio-proxy/0.log"
Oct 31 20:17:47.628944 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:47.628916    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-695c5cddbb-m2rvq_c8f7cf5d-858f-4ae2-b53e-c64881398c3f/thanos-query/0.log"
Oct 31 20:17:47.846282 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:47.846255    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cc4b68895-ftgls_3c19da20-58a9-4a4d-9fe4-7ea13270d595/istio-proxy/0.log"
Oct 31 20:17:48.043182 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:48.043111    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-695c5cddbb-m2rvq_c8f7cf5d-858f-4ae2-b53e-c64881398c3f/kube-rbac-proxy-web/0.log"
Oct 31 20:17:48.460232 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:48.460205    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-695c5cddbb-m2rvq_c8f7cf5d-858f-4ae2-b53e-c64881398c3f/kube-rbac-proxy/0.log"
Oct 31 20:17:48.879349 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:48.879327    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-695c5cddbb-m2rvq_c8f7cf5d-858f-4ae2-b53e-c64881398c3f/prom-label-proxy/0.log"
Oct 31 20:17:49.300799 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:49.300730    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-695c5cddbb-m2rvq_c8f7cf5d-858f-4ae2-b53e-c64881398c3f/kube-rbac-proxy-rules/0.log"
Oct 31 20:17:49.463933 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:49.463907    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55cm8thq_42ddf73a-f3d8-46de-b88f-a5d887e21df8/istio-proxy/0.log"
Oct 31 20:17:49.723628 ip-10-0-132-99 kubenswrapper[5551]: I1031 20:17:49.723595    5551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-695c5cddbb-m2rvq_c8f7cf5d-858f-4ae2-b53e-c64881398c3f/kube-rbac-proxy-metrics/0.log"